Creating a New File for a Dispersed Storage Network

ABSTRACT

A method begins by a dispersed storage (DS) processing module receiving a file creation request to create a new file from portions of a set of existing files. The method continues with the DS processing module identifying data segments associated with the set of existing files corresponding to the portions of the set of existing files to produce identified data segments and identifying sets of encoded data slices of the identified data segments to produce identified sets of encoded data slices. The method continues with the DS processing module determining addressing information for the identified sets of encoded data slices from existing addressing information of the set of existing files and generating a segment allocation table for the new file based on the addressing information of the identified sets of encoded data slices such that the new file is created without duplication of the portions of the existing files.

CROSS REFERENCE TO RELATED PATENTS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. §119(e) to U.S. Provisional Application No. 61/564,185,entitled “Optimizing Performance of Dispersed Storage Network,” filedNov. 28, 2011, pending, which is incorporated herein by reference in itsentirety and made part of the present U.S. Utility patent applicationfor all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

NOT APPLICABLE

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

NOT APPLICABLE

BACKGROUND OF THE INVENTION

1. Technical Field of the Invention

This invention relates generally to computing systems and moreparticularly to data storage solutions within such computing systems.

2. Description of Related Art

Computers are known to communicate, process, and store data. Suchcomputers range from wireless smart phones to data centers that supportmillions of web searches, stock trades, or on-line purchases every day.In general, a computing system generates data and/or manipulates datafrom one form into another. For instance, an image sensor of thecomputing system generates raw picture data and, using an imagecompression program (e.g., JPEG, MPEG, etc.), the computing systemmanipulates the raw picture data into a standardized compressed image.

With continued advances in processing speed and communication speed,computers are capable of processing real time multimedia data forapplications ranging from simple voice communications to streaming highdefinition video. As such, general-purpose information appliances arereplacing purpose-built communications devices (e.g., a telephone). Forexample, smart phones can support telephony communications but they arealso capable of text messaging and accessing the internet to performfunctions including email, web browsing, remote applications access, andmedia communications (e.g., telephony voice, image transfer, musicfiles, video files, real time video streaming. etc.).

Each type of computer is constructed and operates in accordance with oneor more communication, processing, and storage standards. As a result ofstandardization and with advances in technology, more and moreinformation content is being converted into digital formats. Forexample, more digital cameras are now being sold than film cameras, thusproducing more digital pictures. As another example, web-basedprogramming is becoming an alternative to over the air televisionbroadcasts and/or cable broadcasts. As further examples, papers, books,video entertainment, home video, etc. are now being stored digitally,which increases the demand on the storage function of computers.

A typical computer storage system includes one or more memory devicesaligned with the needs of the various operational aspects of thecomputer's processing and communication functions. Generally, theimmediacy of access dictates what type of memory device is used. Forexample, random access memory (RAM) memory can be accessed in any randomorder with a constant response time, thus it is typically used for cachememory and main memory. By contrast, memory device technologies thatrequire physical movement such as magnetic disks, tapes, and opticaldiscs, have a variable response time as the physical movement can takelonger than the data transfer, thus they are typically used forsecondary memory (e.g., hard drive, backup memory, etc.).

A computer's storage system will be compliant with one or more computerstorage standards that include, but are not limited to, network filesystem (NFS), flash file system (FFS), disk file system (DFS), smallcomputer system interface (SCSI), internet small computer systeminterface (iSCSI), file transfer protocol (FTP), and web-baseddistributed authoring and versioning (WebDAV). These standards specifythe data storage format (e.g., files, data objects, data blocks,directories, etc.) and interfacing between the computer's processingfunction and its storage system, which is a primary function of thecomputer's memory controller.

Despite the standardization of the computer and its storage system,memory devices fail; especially commercial grade memory devices thatutilize technologies incorporating physical movement (e.g., a discdrive). For example, it is fairly common for a disc drive to routinelysuffer from bit level corruption and to completely fail after threeyears of use. One solution is to utilize a higher-grade disc drive,which adds significant cost to a computer.

Another solution is to utilize multiple levels of redundant disc drivesto replicate the data into two or more copies. One such redundant driveapproach is called redundant array of independent discs (RAID). In aRAID device, a RAID controller adds parity data to the original databefore storing it across the array. The parity data is calculated fromthe original data such that the failure of a disc will not result in theloss of the original data. For example, RAID 5 uses three discs toprotect data from the failure of a single disc. The parity data, andassociated redundancy overhead data, reduces the storage capacity ofthree independent discs by one third (e.g., n−1=capacity). RAID 6 canrecover from a loss of two discs and requires a minimum of four discswith a storage capacity of n−2.

While RAID addresses the memory device failure issue, it is not withoutits own failures issues that affect its effectiveness, efficiency andsecurity. For instance, as more discs are added to the array, theprobability of a disc failure increases, which increases the demand formaintenance. For example, when a disc fails, it needs to be manuallyreplaced before another disc fails and the data stored in the RAIDdevice is lost. To reduce the risk of data loss, data on a RAID deviceis typically copied on to one or more other RAID devices. While thisaddresses the loss of data issue, it raises a security issue sincemultiple copies of data are available, which increases the chances ofunauthorized access. Further, as the amount of data being stored grows,the overhead of RAID devices becomes a non-trivial efficiency issue.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a computingsystem in accordance with the present invention;

FIG. 2 is a schematic block diagram of an embodiment of a computing corein accordance with the present invention;

FIG. 3 is a schematic block diagram of an embodiment of a distributedstorage processing unit in accordance with the present invention;

FIG. 4 is a schematic block diagram of an embodiment of a grid module inaccordance with the present invention;

FIG. 5 is a diagram of an example embodiment of error coded data slicecreation in accordance with the present invention;

FIG. 6A is a schematic block diagram of an embodiment of a storagemodule in accordance with the present invention;

FIG. 6B is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 6C is a flowchart illustrating an example of prioritizing messagesin accordance with the present invention;

FIG. 7 is a flowchart illustrating an example of modifying a writesequence in accordance with the present invention;

FIG. 8 is a flowchart illustrating another example of modifying a writesequence in accordance with the present invention;

FIG. 9 is a flowchart illustrating another example of modifying a writesequence in accordance with the present invention;

FIG. 10 is a flowchart illustrating an example of retrieving data from adispersed storage network in accordance with the present invention;

FIG. 11 is a diagram illustrating a directory and segment allocationtable structure in accordance with the present invention;

FIG. 12A is a diagram illustrating another directory and segmentallocation table structure in accordance with the present invention;

FIG. 12B is a flowchart illustrating an example of stitching data storedin a dispersed storage network in accordance with the present invention;

FIG. 13A is a diagram illustrating another directory and segmentallocation table structure in accordance with the present invention;

FIG. 13B is a flowchart illustrating an example of splitting data storedin a dispersed storage network in accordance with the present invention;

FIG. 13C is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 13D is a flowchart illustrating an example of creating a new filein accordance with the present invention;

FIG. 14A is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 14B is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 14C is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 15 is a flowchart illustrating an example of detecting a failingmemory in accordance with the present invention;

FIG. 16 is a flowchart illustrating another example of detecting afailing memory in accordance with the present invention;

FIG. 17A is a flowchart illustrating an example of migrating slices inaccordance with the present invention;

FIG. 17B is a diagram of an embodiment of a distributed storage network(DSN) address mapping in accordance with the present invention;

FIG. 17C is a diagram of another embodiment of a distributed storagenetwork (DSN) address mapping in accordance with the present invention;

FIG. 17D is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 17E is a flowchart illustrating another example of migrating slicesin accordance with the present invention;

FIG. 18A is a flowchart illustrating an example of sending a securemessage in accordance with the present invention;

FIG. 18B is a flowchart illustrating an example of processing a securemessage in accordance with the present invention;

FIG. 19A is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 19B is a flowchart illustrating an example of deleting slices inaccordance with the present invention;

FIG. 19C is a schematic block diagram of another embodiment of acomputing system in accordance with the present invention;

FIG. 19D is a flowchart illustrating another example of deleting slicesin accordance with the present invention;

FIG. 20 is a flowchart illustrating an example of outputting data inaccordance with the present invention; and

FIG. 21 is a flowchart illustrating an example of inputting data inaccordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic block diagram of a computing system 10 thatincludes one or more of a first type of user devices 12, one or more ofa second type of user devices 14, at least one distributed storage (DS)processing unit 16, at least one DS managing unit 18, at least onestorage integrity processing unit 20, and a distributed storage network(DSN) memory 22 coupled via a network 24. The network 24 may include oneor more wireless and/or wire lined communication systems; one or moreprivate intranet systems and/or public internet systems; and/or one ormore local area networks (LAN) and/or wide area networks (WAN).

The DSN memory 22 includes a plurality of distributed storage (DS) units36 for storing data of the system. Each of the DS units 36 includes aprocessing module and memory and may be located at a geographicallydifferent site than the other DS units (e.g., one in Chicago, one inMilwaukee, etc.).

Each of the user devices 12-14, the DS processing unit 16, the DSmanaging unit 18, and the storage integrity processing unit 20 may be aportable computing device (e.g., a social networking device, a gamingdevice, a cell phone, a smart phone, a personal digital assistant, adigital music player, a digital video player, a laptop computer, ahandheld computer, a video game controller, and/or any other portabledevice that includes a computing core) and/or a fixed computing device(e.g., a personal computer, a computer server, a cable set-top box, asatellite receiver, a television set, a printer, a fax machine, homeentertainment equipment, a video game console, and/or any type of homeor office computing equipment). Such a portable or fixed computingdevice includes a computing core 26 and one or more interfaces 30, 32,and/or 33. An embodiment of the computing core 26 will be described withreference to FIG. 2.

With respect to the interfaces, each of the interfaces 30, 32, and 33includes software and/or hardware to support one or more communicationlinks via the network 24 indirectly and/or directly. For example,interfaces 30 support a communication link (wired, wireless, direct, viaa LAN, via the network 24, etc.) between the first type of user device14 and the DS processing unit 16. As another example, DSN interface 32supports a plurality of communication links via the network 24 betweenthe DSN memory 22 and the DS processing unit 16, the first type of userdevice 12, and/or the storage integrity processing unit 20. As yetanother example, interface 33 supports a communication link between theDS managing unit 18 and any one of the other devices and/or units 12,14, 16, 20, and/or 22 via the network 24.

In general and with respect to data storage, the system 10 supportsthree primary functions: distributed network data storage management,distributed data storage and retrieval, and data storage integrityverification. In accordance with these three primary functions, data canbe distributedly stored in a plurality of physically different locationsand subsequently retrieved in a reliable and secure manner regardless offailures of individual storage devices, failures of network equipment,the duration of storage, the amount of data being stored, attempts athacking the data, etc.

The DS managing unit 18 performs distributed network data storagemanagement functions, which include establishing distributed datastorage parameters, performing network operations, performing networkadministration, and/or performing network maintenance. The DS managingunit 18 establishes the distributed data storage parameters (e.g.,allocation of virtual DSN memory space, distributed storage parameters,security parameters, billing information, user profile information,etc.) for one or more of the user devices 12-14 (e.g., established forindividual devices, established for a user group of devices, establishedfor public access by the user devices, etc.). For example, the DSmanaging unit 18 coordinates the creation of a vault (e.g., a virtualmemory block) within the DSN memory 22 for a user device (for a group ofdevices, or for public access). The DS managing unit 18 also determinesthe distributed data storage parameters for the vault. In particular,the DS managing unit 18 determines a number of slices (e.g., the numberthat a data segment of a data file and/or data block is partitioned intofor distributed storage) and a read threshold value (e.g., the minimumnumber of slices required to reconstruct the data segment).

As another example, the DS managing module 18 creates and stores,locally or within the DSN memory 22, user profile information. The userprofile information includes one or more of authentication information,permissions, and/or the security parameters. The security parameters mayinclude one or more of encryption/decryption scheme, one or moreencryption keys, key generation scheme, and data encoding/decodingscheme.

As yet another example, the DS managing unit 18 creates billinginformation for a particular user, user group, vault access, publicvault access, etc. For instance, the DS managing unit 18 tracks thenumber of times user accesses a private vault and/or public vaults,which can be used to generate a per-access bill. In another instance,the DS managing unit 18 tracks the amount of data stored and/orretrieved by a user device and/or a user group, which can be used togenerate a per-data-amount bill.

The DS managing unit 18 also performs network operations, networkadministration, and/or network maintenance. As at least part ofperforming the network operations and/or administration, the DS managingunit 18 monitors performance of the devices and/or units of the system10 for potential failures, determines the devices and/or unit'sactivation status, determines the devices' and/or units' loading, andany other system level operation that affects the performance level ofthe system 10. For example, the DS managing unit 18 receives andaggregates network management alarms, alerts, errors, statusinformation, performance information, and messages from the devices12-14 and/or the units 16, 20, 22. For example, the DS managing unit 18receives a simple network management protocol (SNMP) message regardingthe status of the DS processing unit 16.

The DS managing unit 18 performs the network maintenance by identifyingequipment within the system 10 that needs replacing, upgrading,repairing, and/or expanding. For example, the DS managing unit 18determines that the DSN memory 22 needs more DS units 36 or that one ormore of the DS units 36 needs updating.

The second primary function (i.e., distributed data storage andretrieval) begins and ends with a user device 12-14. For instance, if asecond type of user device 14 has a data file 38 and/or data block 40 tostore in the DSN memory 22, it send the data file 38 and/or data block40 to the DS processing unit 16 via its interface 30. As will bedescribed in greater detail with reference to FIG. 2, the interface 30functions to mimic a conventional operating system (OS) file systeminterface (e.g., network file system (NFS), flash file system (FFS),disk file system (DFS), file transfer protocol (FTP), web-baseddistributed authoring and versioning (WebDAV), etc.) and/or a blockmemory interface (e.g., small computer system interface (SCSI), internetsmall computer system interface (iSCSI), etc.). In addition, theinterface 30 may attach a user identification code (ID) to the data file38 and/or data block 40.

The DS processing unit 16 receives the data file 38 and/or data block 40via its interface 30 and performs a distributed storage (DS) process 34thereon (e.g., an error coding dispersal storage function). The DSprocessing 34 begins by partitioning the data file 38 and/or data block40 into one or more data segments, which is represented as Y datasegments. For example, the DS processing 34 may partition the data file38 and/or data block 40 into a fixed byte size segment (e.g., 2¹ to2^(n) bytes, where n=>2) or a variable byte size (e.g., change byte sizefrom segment to segment, or from groups of segments to groups ofsegments, etc.).

For each of the Y data segments, the DS processing 34 error encodes(e.g., forward error correction (FEC), information dispersal algorithm,or error correction coding) and slices (or slices then error encodes)the data segment into a plurality of error coded (EC) data slices 42-48,which is represented as X slices per data segment. The number of slices(X) per segment, which corresponds to a number of pillars n, is set inaccordance with the distributed data storage parameters and the errorcoding scheme. For example, if a Reed-Solomon (or other FEC scheme) isused in an n/k system, then a data segment is divided into n slices,where k number of slices is needed to reconstruct the original data(i.e., k is the threshold). As a few specific examples, the n/k factormay be 5/3; 6/4; 8/6; 8/5; 16/10.

For each slice 42-48, the DS processing unit 16 creates a unique slicename and appends it to the corresponding slice 42-48. The slice nameincludes universal DSN memory addressing routing information (e.g.,virtual memory addresses in the DSN memory 22) and user-specificinformation (e.g., user ID, file name, data block identifier, etc.).

The DS processing unit 16 transmits the plurality of EC slices 42-48 toa plurality of DS units 36 of the DSN memory 22 via the DSN interface 32and the network 24. The DSN interface 32 formats each of the slices fortransmission via the network 24. For example, the DSN interface 32 mayutilize an internet protocol (e.g., TCP/IP, etc.) to packetize theslices 42-48 for transmission via the network 24.

The number of DS units 36 receiving the slices 42-48 is dependent on thedistributed data storage parameters established by the DS managing unit18. For example, the DS managing unit 18 may indicate that each slice isto be stored in a different DS unit 36. As another example, the DSmanaging unit 18 may indicate that like slice numbers of different datasegments are to be stored in the same DS unit 36. For example, the firstslice of each of the data segments is to be stored in a first DS unit36, the second slice of each of the data segments is to be stored in asecond DS unit 36, etc. In this manner, the data is encoded anddistributedly stored at physically diverse locations to improved datastorage integrity and security.

Each DS unit 36 that receives a slice 42-48 for storage translates thevirtual DSN memory address of the slice into a local physical addressfor storage. Accordingly, each DS unit 36 maintains a virtual tophysical memory mapping to assist in the storage and retrieval of data.

The first type of user device 12 performs a similar function to storedata in the DSN memory 22 with the exception that it includes the DSprocessing. As such, the device 12 encodes and slices the data fileand/or data block it has to store. The device then transmits the slices11 to the DSN memory via its DSN interface 32 and the network 24.

For a second type of user device 14 to retrieve a data file or datablock from memory, it issues a read command via its interface 30 to theDS processing unit 16. The DS processing unit 16 performs the DSprocessing 34 to identify the DS units 36 storing the slices of the datafile and/or data block based on the read command. The DS processing unit16 may also communicate with the DS managing unit 18 to verify that theuser device 14 is authorized to access the requested data.

Assuming that the user device is authorized to access the requesteddata, the DS processing unit 16 issues slice read commands to at least athreshold number of the DS units 36 storing the requested data (e.g., toat least 10 DS units for a 16/10 error coding scheme). Each of the DSunits 36 receiving the slice read command, verifies the command,accesses its virtual to physical memory mapping, retrieves the requestedslice, or slices, and transmits it to the DS processing unit 16.

Once the DS processing unit 16 has received a read threshold number ofslices for a data segment, it performs an error decoding function andde-slicing to reconstruct the data segment. When Y number of datasegments has been reconstructed, the DS processing unit 16 provides thedata file 38 and/or data block 40 to the user device 14. Note that thefirst type of user device 12 performs a similar process to retrieve adata file and/or data block.

The storage integrity processing unit 20 performs the third primaryfunction of data storage integrity verification. In general, the storageintegrity processing unit 20 periodically retrieves slices 45, and/orslice names, of a data file or data block of a user device to verifythat one or more slices have not been corrupted or lost (e.g., the DSunit failed). The retrieval process mimics the read process previouslydescribed.

If the storage integrity processing unit 20 determines that one or moreslices is corrupted or lost, it rebuilds the corrupted or lost slice(s)in accordance with the error coding scheme. The storage integrityprocessing unit 20 stores the rebuild slice, or slices, in theappropriate DS unit(s) 36 in a manner that mimics the write processpreviously described.

FIG. 2 is a schematic block diagram of an embodiment of a computing core26 that includes a processing module 50, a memory controller 52, mainmemory 54, a video graphics processing unit 55, an input/output (IO)controller 56, a peripheral component interconnect (PCI) interface 58,at least one IO device interface module 62, a read only memory (ROM)basic input output system (BIOS) 64, and one or more memory interfacemodules. The memory interface module(s) includes one or more of auniversal serial bus (USB) interface module 66, a host bus adapter (HBA)interface module 68, a network interface module 70, a flash interfacemodule 72, a hard drive interface module 74, and a DSN interface module76. Note the DSN interface module 76 and/or the network interface module70 may function as the interface 30 of the user device 14 of FIG. 1.Further note that the IO device interface module 62 and/or the memoryinterface modules may be collectively or individually referred to as IOports.

FIG. 3 is a schematic block diagram of an embodiment of a dispersedstorage (DS) processing module 34 of user device 12 and/or of the DSprocessing unit 16. The DS processing module 34 includes a gatewaymodule 78, an access module 80, a grid module 82, and a storage module84. The DS processing module 34 may also include an interface 30 and theDSnet interface 32 or the interfaces 68 and/or 70 may be part of user 12or of the DS processing unit 16. The DS processing module 34 may furtherinclude a bypass/feedback path between the storage module 84 to thegateway module 78. Note that the modules 78-84 of the DS processingmodule 34 may be in a single unit or distributed across multiple units.

In an example of storing data, the gateway module 78 receives anincoming data object that includes a user ID field 86, an object namefield 88, and the data field 40 and may also receive correspondinginformation that includes a process identifier (e.g., an internalprocess/application ID), metadata, a file system directory, a blocknumber, a transaction message, a user device identity (ID), a dataobject identifier, a source name, and/or user information. The gatewaymodule 78 authenticates the user associated with the data object byverifying the user ID 86 with the managing unit 18 and/or anotherauthenticating unit.

When the user is authenticated, the gateway module 78 obtains userinformation from the management unit 18, the user device, and/or theother authenticating unit. The user information includes a vaultidentifier, operational parameters, and user attributes (e.g., userdata, billing information, etc.). A vault identifier identifies a vault,which is a virtual memory space that maps to a set of DS storage units36. For example, vault 1 (i.e., user 1's DSN memory space) includeseight DS storage units (X=8 wide) and vault 2 (i.e., user 2's DSN memoryspace) includes sixteen DS storage units (X=16 wide). The operationalparameters may include an error coding algorithm, the width n (number ofpillars X or slices per segment for this vault), a read threshold T, awrite threshold, an encryption algorithm, a slicing parameter, acompression algorithm, an integrity check method, caching settings,parallelism settings, and/or other parameters that may be used to accessthe DSN memory layer.

The gateway module 78 uses the user information to assign a source name35 to the data. For instance, the gateway module 78 determines thesource name 35 of the data object 40 based on the vault identifier andthe data object. For example, the source name may contain a fileidentifier (ID), a vault generation number, a reserved field, and avault identifier (ID). As another example, the gateway module 78 maygenerate the file ID based on a hash function of the data object 40.Note that the gateway module 78 may also perform message conversion,protocol conversion, electrical conversion, optical conversion, accesscontrol, user identification, user information retrieval, trafficmonitoring, statistics generation, configuration, management, and/orsource name determination.

The access module 80 receives the data object 40 and creates a series ofdata segments 1 through Y 90-92 in accordance with a data storageprotocol (e.g., file storage system, a block storage system, and/or anaggregated block storage system). The number of segments Y may be chosenor randomly assigned based on a selected segment size and the size ofthe data object. For example, if the number of segments is chosen to bea fixed number, then the size of the segments varies as a function ofthe size of the data object. For instance, if the data object is animage file of 4,194,304 eight bit bytes (e.g., 33,554,432 bits) and thenumber of segments Y=131,072, then each segment is 256 bits or 32 bytes.As another example, if segment sized is fixed, then the number ofsegments Y varies based on the size of data object. For instance, if thedata object is an image file of 4,194,304 bytes and the fixed size ofeach segment is 4,096 bytes, the then number of segments Y=1,024. Notethat each segment is associated with the same source name.

The grid module 82 receives the data segments and may manipulate (e.g.,compression, encryption, cyclic redundancy check (CRC), etc.) each ofthe data segments before performing an error coding function of theerror coding dispersal storage function to produce a pre-manipulateddata segment. After manipulating a data segment, if applicable, the gridmodule 82 error encodes (e.g., Reed-Solomon, Convolution encoding,Trellis encoding, etc.) the data segment or manipulated data segmentinto X error coded data slices 42-44.

The value X, or the number of pillars (e.g., X=16), is chosen as aparameter of the error coding dispersal storage function. Otherparameters of the error coding dispersal function include a readthreshold T, a write threshold W, etc. The read threshold (e.g., T=10,when X=16) corresponds to the minimum number of error-free error codeddata slices required to reconstruct the data segment. In other words,the DS processing module 34 can compensate for X-T (e.g., 16−10=6)missing error coded data slices per data segment. The write threshold Wcorresponds to a minimum number of DS storage units that acknowledgeproper storage of their respective data slices before the DS processingmodule indicates proper storage of the encoded data segment. Note thatthe write threshold is greater than or equal to the read threshold for agiven number of pillars (X).

For each data slice of a data segment, the grid module 82 generates aunique slice name 37 and attaches it thereto. The slice name 37 includesa universal routing information field and a vault specific field and maybe 48 bytes (e.g., 24 bytes for each of the universal routinginformation field and the vault specific field). As illustrated, theuniversal routing information field includes a slice index, a vault ID,a vault generation, and a reserved field. The slice index is based onthe pillar number and the vault ID and, as such, is unique for eachpillar (e.g., slices of the same pillar for the same vault for anysegment will share the same slice index). The vault specific fieldincludes a data name, which includes a file ID and a segment number(e.g., a sequential numbering of data segments 1-Y of a simple dataobject or a data block number).

Prior to outputting the error coded data slices of a data segment, thegrid module may perform post-slice manipulation on the slices. Ifenabled, the manipulation includes slice level compression, encryption,CRC, addressing, tagging, and/or other manipulation to improve theeffectiveness of the computing system.

When the error coded data slices of a data segment are ready to beoutputted, the grid module 82 determines which of the DS storage units36 will store the EC data slices based on a dispersed storage memorymapping associated with the user's vault and/or DS storage unitattributes. The DS storage unit attributes may include availability,self-selection, performance history, link speed, link latency,ownership, available DSN memory, domain, cost, a prioritization scheme,a centralized selection message from another source, a lookup table,data ownership, and/or any other factor to optimize the operation of thecomputing system. Note that the number of DS storage units 36 is equalto or greater than the number of pillars (e.g., X) so that no more thanone error coded data slice of the same data segment is stored on thesame DS storage unit 36. Further note that EC data slices of the samepillar number but of different segments (e.g., EC data slice 1 of datasegment 1 and EC data slice 1 of data segment 2) may be stored on thesame or different DS storage units 36.

The storage module 84 performs an integrity check on the outboundencoded data slices and, when successful, identifies a plurality of DSstorage units based on information provided by the grid module 82. Thestorage module 84 then outputs the encoded data slices 1 through X ofeach segment 1 through Y to the DS storage units 36. Each of the DSstorage units 36 stores its EC data slice(s) and maintains a localvirtual DSN address to physical location table to convert the virtualDSN address of the EC data slice(s) into physical storage addresses.

In an example of a read operation, the user device 12 and/or 14 sends aread request to the DS processing unit 16, which authenticates therequest. When the request is authentic, the DS processing unit 16 sendsa read message to each of the DS storage units 36 storing slices of thedata object being read. The slices are received via the DSnet interface32 and processed by the storage module 84, which performs a parity checkand provides the slices to the grid module 82 when the parity check wassuccessful. The grid module 82 decodes the slices in accordance with theerror coding dispersal storage function to reconstruct the data segment.The access module 80 reconstructs the data object from the data segmentsand the gateway module 78 formats the data object for transmission tothe user device.

FIG. 4 is a schematic block diagram of an embodiment of a grid module 82that includes a control unit 73, a pre-slice manipulator 75, an encoder77, a slicer 79, a post-slice manipulator 81, a pre-slice de-manipulator83, a decoder 85, a de-slicer 87, and/or a post-slice de-manipulator 89.Note that the control unit 73 may be partially or completely external tothe grid module 82. For example, the control unit 73 may be part of thecomputing core at a remote location, part of a user device, part of theDS managing unit 18, or distributed amongst one or more DS storageunits.

In an example of write operation, the pre-slice manipulator 75 receivesa data segment 90-92 and a write instruction from an authorized userdevice. The pre-slice manipulator 75 determines if pre-manipulation ofthe data segment 90-92 is required and, if so, what type. The pre-slicemanipulator 75 may make the determination independently or based oninstructions from the control unit 73, where the determination is basedon a computing system-wide predetermination, a table lookup, vaultparameters associated with the user identification, the type of data,security requirements, available DSN memory, performance requirements,and/or other metadata.

Once a positive determination is made, the pre-slice manipulator 75manipulates the data segment 90-92 in accordance with the type ofmanipulation. For example, the type of manipulation may be compression(e.g., Lempel-Ziv-Welch, Huffman, Golomb, fractal, wavelet, etc.),signatures (e.g., Digital Signature Algorithm (DSA), Elliptic Curve DSA,Secure Hash Algorithm, etc.), watermarking, tagging, encryption (e.g.,Data Encryption Standard, Advanced Encryption Standard, etc.), addingmetadata (e.g., time/date stamping, user information, file type, etc.),cyclic redundancy check (e.g., CRC32), and/or other data manipulationsto produce the pre-manipulated data segment.

The encoder 77 encodes the pre-manipulated data segment 92 using aforward error correction (FEC) encoder (and/or other type of erasurecoding and/or error coding) to produce an encoded data segment 94. Theencoder 77 determines which forward error correction algorithm to usebased on a predetermination associated with the user's vault, a timebased algorithm, user direction, DS managing unit direction, controlunit direction, as a function of the data type, as a function of thedata segment 92 metadata, and/or any other factor to determine algorithmtype. The forward error correction algorithm may be Golay,Multidimensional parity, Reed-Solomon, Hamming, Bose Ray ChauduriHocquenghem (BCH), Cauchy-Reed-Solomon, or any other FEC encoder. Notethat the encoder 77 may use a different encoding algorithm for each datasegment 92, the same encoding algorithm for the data segments 92 of adata object, or a combination thereof.

The encoded data segment 94 is of greater size than the data segment 92by the overhead rate of the encoding algorithm by a factor of X/T, whereX is the width or number of slices, and T is the read threshold. In thisregard, the corresponding decoding process can accommodate at most X-Tmissing EC data slices and still recreate the data segment 92. Forexample, if X=16 and T=10, then the data segment 92 will be recoverableas long as 10 or more EC data slices per segment are not corrupted.

The slicer 79 transforms the encoded data segment 94 into EC data slicesin accordance with the slicing parameter from the vault for this userand/or data segment 92. For example, if the slicing parameter is X=16,then the slicer 79 slices each encoded data segment 94 into 16 encodedslices.

The post-slice manipulator 81 performs, if enabled, post-manipulation onthe encoded slices to produce the EC data slices. If enabled, thepost-slice manipulator 81 determines the type of post-manipulation,which may be based on a computing system-wide predetermination,parameters in the vault for this user, a table lookup, the useridentification, the type of data, security requirements, available DSNmemory, performance requirements, control unit directed, and/or othermetadata. Note that the type of post-slice manipulation may includeslice level compression, signatures, encryption, CRC, addressing,watermarking, tagging, adding metadata, and/or other manipulation toimprove the effectiveness of the computing system.

In an example of a read operation, the post-slice de-manipulator 89receives at least a read threshold number of EC data slices and performsthe inverse function of the post-slice manipulator 81 to produce aplurality of encoded slices. The de-slicer 87 de-slices the encodedslices to produce an encoded data segment 94. The decoder 85 performsthe inverse function of the encoder 77 to recapture the data segment90-92. The pre-slice de-manipulator 83 performs the inverse function ofthe pre-slice manipulator 75 to recapture the data segment 90-92.

FIG. 5 is a diagram of an example of slicing an encoded data segment 94by the slicer 79. In this example, the encoded data segment 94 includesthirty-two bits, bytes, data words, etc., but may include more or lessbits, bytes, data words, etc. The slicer 79 disperses the bits of theencoded data segment 94 across the EC data slices in a pattern as shown.As such, each EC data slice does not include consecutive bits, bytes,data words, etc. of the data segment 94 reducing the impact ofconsecutive bit, byte, data word, etc. failures on data recovery. Forexample, if EC data slice 2 (which includes bits 1, 5, 9, 13, 17, 25,and 29) is unavailable (e.g., lost, inaccessible, or corrupted), thedata segment can be reconstructed from the other EC data slices (e.g.,1, 3 and 4 for a read threshold of 3 and a width of 4).

FIG. 6A is a schematic block diagram of an embodiment of a storagemodule 84 that includes a writer 102, a reader 104, and queues 1-5. Thewriter 102 generates messages for transmission to one or more DS unitsof DS units 1-5. The reader 104 interprets messages received from theone or more DS units of DS units 1-5. The messages include requestmessages and response messages. Messages transmitted from the storagemodule 84 to DS units 1-5 include requests 1-5. Messages that thestorage module 84 receives from DS units 1-5 include responses 1-5.

Each queue of queues 1-5 may be implemented as one or more of a physicalmemory device, a plurality of memory devices, and a virtual allocationof storage capacity of one or more memory devices. Each queue may beassociated with a fixed storage capacity. Each queue of queues 1-5temporarily stores messages received from a DS unit waiting to beprocessed by the reader 104 or messages from the writer 102 to betransmitted to a DS unit. For example, the writer 102 stores message 3-5in queue 3 for transmission to DS unit 3. Message 3-5 are sent to DSunit 3 via a network when message 3-5 are to be transmitted inaccordance with a queue prioritization scheme.

The queue prioritization scheme may be based on one or more of a numberof messages associated with the queue (e.g., pending messages), aprioritization approach (e.g., first in first out (FIFO), last in lastout (LIFO)), a prioritization level associated with each of the messagesassociated with the queue, a network performance level, a DS unitperformance level, and an order of message receipt by the queue. Forinstance, queue 3 outputs message 3-5 to DS unit 3 when messages 3-1through 3-4 have been successfully sent in accordance with a FIFOprioritization approach of the queue prioritization scheme. As anotherinstance, queue 3 outputs message 3-5 to DS unit 3 when message 3-1 hasbeen successfully sent and prior to sending of messages 3-2 through 3-4when a prioritization level associated with message 3-5 is greater thana privatization level associated with messages 3-2 through 3-4 and theprioritization level associated with message 3-5 is lower than aprivatization level associated with message 3-1. As another example,queue 4 receives message 4-0 from DS unit 4 to be read by the reader104. Queue 4 outputs message 4-0 to the reader 104 in accordance withthe queue prioritization scheme.

The storage module 84 may delete a message stored in a queue when themessage is outputted and is no longer required. The storage module 84may change a message priority level of the message after message hasbeen stored in a queue to affect a modified message transmission order.The storage module 84 may delete the message stored in the queue whenthe message is no longer required. The method to determine whether themessage is no longer required, to delete the message, and to change themessage priority is discussed in greater detail with reference to FIGS.6B-10.

FIG. 6B is a schematic block diagram of another embodiment of acomputing system that includes a computing device 110 and a dispersedstorage network (DSN) memory 22 of a dispersed storage network. The DSNmemory 22 includes a plurality of storage nodes 112-116. The computingdevice 110 includes a dispersed storage (DS) processing 118. Thecomputing device 110 may be implemented as at least one of a userdevice, a DS processing unit, and a DS unit. The DS processing 118includes a generate messages module 120, a processing information module122, a prioritization module 124, and a messaging module 126. The systemfunctions to access the DSN memory 22 with regards to a set of encodeddata slices. The accessing includes at least one of reading the set ofencoded data slices from the DSN memory 22 and writing the set ofencoded data slices to the DSN memory 22. A data segment 128 of data isencoded using a dispersed storage error coding function to produce theset of encoded data slices. The generate messages module 120 receivesthe data segment 128 when the accessing includes writing the set ofencoded data slices to the DSN memory 22. For example, the generatemessages module 120 receives the data segment 128 and encodes the datasegment 128 to produce the set of encoded data slices when the accessingincludes writing the set of encoded data slices to the DSN memory 22. Asanother example, the DS processing 118 generates the data segment 128and encodes the data segment 128 to produce the set of encoded dataslices when the accessing includes writing the set of encoded dataslices to the DSN memory 22. Alternatively, the generate messages module120 receives the set of encoded data slices.

The generate messages module 120 generates a set of messages 130regarding the set of encoded data slices. The set of messages 130includes a set of read messages to read the set of encoded data slicesfrom the DSN memory 22 when the accessing includes reading the set ofencoded data slices from the DSN memory 22. A read message of the set ofread messages includes a read slice request. For example, the generatemessages module 120 generates a set of read slice requests that includesa set of slice names corresponding to the set of encoded data slices.The set of messages 130 includes a set of write messages to write theset of encoded data slices to the DSN memory 22 when the accessingincludes writing the set of encoded data slices to the DSN memory 22. Awrite message of the set of write messages includes a write slicerequest. For example, the generate messages module 120 generates a setof write slice requests that includes the set of encoded data slices andthe set of slice names corresponding to the set of encoded data slices.

The processing information module 122 determines system-level messageprocessing information 132 based on status of processing a plurality ofsets of messages regarding a plurality of sets of encoded data slices.The plurality of sets of messages regarding the plurality of sets ofencoded data slices may include numerous other write and read accessesof other encoded data slices within at least one storage node of theplurality of storage nodes 112-116. The processing information module122 determines the system-level message processing information 132 by aseries of steps. A first step includes, for a first set of messages ofthe plurality of sets of messages, determining at least one of: acurrent status of sending the first set of messages (e.g., the first setof messages have been sent to the DSN memory 22), and a current statusof successfully processing the first set of messages. The determiningincludes at least one of initiating a query, performing a lookup,executing a test, accessing historical records, and accessing themessaging module 126. The processing of the first set of messagesincludes at least one of retrieving and writing the set of encoded dataslices. For example, a current status of sending the first set ofmessages indicates that 5 messages of a set of 16 messages have beensent. Successfully processing the first set of messages may include atleast one of retrieving at least a decode threshold number of encodeddata slices of the set of encoded data slices and writing at least awrite threshold number of encoded data slices of the set of encoded dataslices. For example, a current status of successfully processing thefirst set of messages indicates successful processing when 11 encodeddata slices have been retrieved when a decode threshold number is 10. Asanother example, a current status of successfully processing the firstset of messages indicates unsuccessful processing when 12 encoded dataslices have been sent when a write threshold number is 13.

A second step of determining the system-level message processinginformation includes, for a second set of messages of the plurality ofsets of messages, determining at least one of: a current status ofsending the second set of messages, and a current status of successfullyprocessing the second set of messages. Alternatively, or in addition to,more steps may be included in the series of steps including determiningstatus with regards to further sets of messages of the plurality of setsof messages. A third step includes, determining the status of processingthe plurality of sets of messages regarding the plurality of sets ofencoded data slices based on the at least one of the current status ofsending the first set of messages and the current status of successfullyprocessing of the first set of messages, and the at least one of thecurrent status of sending the second set of messages, and the currentstatus of successfully processing of the second set of messages. Thedetermining includes at least one of aggregating status, selectingstatus, and confirming status. For example, the processing informationmodule 122 determines the status of processing the plurality sets ofmessages regarding the plurality of sets of encoded data slices byaggregating current status associated with 10 sets of messages when theplurality of sets of messages includes 10 sets of messages.

For a first message 136 of the set of messages 130, the prioritizationmodule 124 determines a first message priority 134 based on thesystem-level message processing information 132 and message processingstatus of a first storage node 112 of the plurality of storage nodes112-116. The prioritization module 124 determines the message processingstatus of the first storage node 112 by a series of steps. A first stepincludes determining a number of sets of the plurality of sets ofmessages that involves the first storage node (e.g., messages to be sentto the first storage node 112). A second step includes determiningstatus of sending messages of the number of sets of the plurality ofsets of messages to the first storage node 112. A third step includesdetermining status of successfully processed messages of the number ofsets of the plurality of sets of messages by the first storage node 112.A fourth step includes determining the message processing status of thefirst storage node 112 based on the status of sending messages and thestatus of successfully processed messages.

The prioritization module 124 determines the first message priority 134by a series of steps. A first step includes determining the number ofsets of the plurality of sets of messages that involves the firststorage node 112. A second step includes interpreting the system-levelmessage processing information 132 regarding the number of sets thatinvolve the first storage node 112 to produce interpreted system-levelmessage processing information. A third step includes interpreting themessage processing status of the first storage node 112 regarding thenumber of sets that involve the first storage node 112 to produceinterpreted message processing status. A fourth step includes applying aload balancing function in accordance the interpreted system-levelmessage processing information and the interpreted message processingstatus to produce the first message priority 134. The load balancingfunction includes at least one of a first in first out function, a lastin first out function, a time-based function, and a threshold-basedfunction. For example, the prioritization module 124 produces the firstmessage priority 134 to include a lower than average priority level whenthe interpreted system-level message processing information indicatesthat a plurality of other messages are pending to be sent to the firststorage node 112 where the plurality of other messages are associatedwith sets of encoded data slices that have not achieved processing of athreshold number of each set of the sets of encoded data slices. Themessaging module 126 sends the first message 136 of the set of messages130 to the first storage node 112 in accordance with the first messagepriority 134. For example, the messaging module 126 sends the firstmessage 136 to the first storage node 112 subsequent to sending anothermessage to the first storage node 112 when a message priority of theother message has greater priority than priority of the first messagepriority 134.

For a second message 140 of the set of messages 130, the prioritizationmodule 124 determines a second message priority 138 based on thesystem-level message processing information 132 and message processingstatus of a second storage node 114 of the plurality of storage nodes112-116. The messaging module 126 sends the second message 140 of theset of messages 130 to the second storage node 114 in accordance withthe second message priority 138. For example, the messaging module 126sends the second message 140 to the second storage node 114 prior tosending a different message to the second storage node 114 when amessage priority of the different message has less priority thanpriority of the second message priority 138.

The system further functions to update message priorities. Theprioritization module 124 updates the first message priority 134 basedon status of processing the set of messages 130. The prioritizationmodule 124 updates the second message priority 138 based on status ofprocessing the set of messages 130. The updating of message priorityincludes updating message priority associated with writing the set ofencoded data slices to the DSN memory 22 and reading the set of encodeddata slices from the DSN memory 22. The status of processing the sets ofmessages 130 includes status with regards to at least one of a number ofmessages of the set of messages that have been sent and/or processed anda number of messages of the set of messages that have been sent and/orprocessed within a given time period.

When the set of messages 130 includes the set of write messages to writethe set of encoded data slices to the DSN memory 22, the messagingmodule 126 determines when a write threshold number of the set of writemessages have been sent to the DSN memory 22. When the write thresholdnumber of the set of write messages have been sent and the first message136 has not yet been sent to the first storage node 112, theprioritization module 124 reduces the first message priority 134.Alternatively, the messaging module 126 determines when a writethreshold number of the set of write messages have been successfullyprocessed by the DSN memory 22. When the write threshold number of theset of write messages have been successfully processed and the firstmessage 136 has not yet been sent to the first storage node 112, theprioritization module 124 reduces the first message priority 134.

When the set of messages 130 includes the set of read messages to readthe set of encoded data slices from the DSN memory 22, the messagingmodule 126 determines when a decode threshold number of the set of readmessages have been sent to the DSN memory 22. When the decode thresholdnumber of the set of read messages have been sent and the first message136 has not yet been sent to the first storage node 112, theprioritization module 124 reduces the first message priority 134.Alternatively, the messaging module 126 determines when a decodethreshold number of the set of read messages have been successfullyprocessed by the DSN memory 22. When the decode threshold number of theset of read messages have been successfully processed and the firstmessage 136 has not yet been sent to the first storage node 112, theprioritization module 124 reduces the first message priority 134.

When the set of messages 130 includes the set of write messages to writethe set of encoded data slices to the DSN memory 22, the messagingmodule 126 determines that a write threshold number of the set of writemessages have not been sent to DSN memory 22 within a given time frame.When the write threshold number of the set of write messages have notbeen sent in the given time period and the first message 136 has not yetbeen sent to the first storage node 112 within the given time period,the prioritization module 124 increases the first message priority 134.

Alternatively, when the set of messages 130 includes the set of writemessages to write the set of encoded data slices to the DSN memory 22,the messaging module 126 determines when a write threshold number of theset of write messages have not been successfully processed by the DSNmemory 22 within a given time period. When the write threshold number ofthe set of write messages have not been successfully processed withinthe given time period and the first message 136 has not yet been sent tothe first storage node 112, the prioritization module 124 increases thefirst message priority 134.

When the set of messages includes a set of read messages to read the setof encoded data slices from the DSN memory 22, the messaging module 126determines when a decode threshold number of the set of read messageshave not been sent to the DSN memory 22 within a given time period. Whenthe decode threshold number of the set of read messages have not beensent within the given time period and the first message 136 has not yetbeen sent to the first storage node 112 in the given time period, theprioritization module 124 increases the first message priority 134.

Alternatively, when the set of messages 130 includes the set of readmessages to read the set of encoded data slices from the DSN memory 22,the messaging module 126 determines when a decode threshold number ofthe set of read messages have not been successfully processed by the DSNmemory 22 within a given time period. When the decode threshold numberof the set of read messages have not been successfully processed withinthe given time period and the first message 136 has not yet been sent tothe first storage node 112, the prioritization module 124 increases thefirst message priority 134.

FIG. 6C is a flowchart illustrating an example of prioritizing messages.The method begins at step 150 where a processing module (e.g., of adispersed storage (DS) processing) generates a set of messages regardinga set of encoded data slices. The set of messages may be utilized toaccess a dispersed storage network (DSN) including writing the set ofencoded data slices to the DSN and reading the set of encoded dataslices from the DSN. A data segment of data is encoded using a dispersedstorage error coding function to produce the set of encoded data slices.

The method continues at step 152 where the processing module determinessystem-level message processing information based on status ofprocessing a plurality of sets of messages regarding a plurality of setsof encoded data slices. The determining system-level message processinginformation includes a series of steps. A first step includes, for afirst set of messages of the plurality of sets of messages, determiningat least one of: a current status of sending the first set of messages,and a current status of successfully processing of the first set ofmessages. A second step includes, for a second set of messages of theplurality of sets of messages, determining at least one of: a currentstatus of sending the second set of messages, and a current status ofsuccessfully processing of the second set of messages. A third stepincludes determining the status of processing the plurality of sets ofmessages regarding the plurality of sets of encoded data slices based onthe at least one of the current status of sending the first set ofmessages and the current status of successfully processing of the firstset of messages, and the at least one of the current status of sendingthe second set of messages, and the current status of successfullyprocessing of the second set of messages.

For a first message of the set of messages, the method continues at step154 where the processing module determines a first message prioritybased on the system-level message processing information and messageprocessing status of a first storage node of the DSN. The processingmodule determines the message processing status of the first storagenode by a series of steps. A first step includes determining a number ofsets of the plurality of sets of messages that involves the firststorage node. A second step includes determining status of sendingmessages of the number of sets of the plurality of sets of messages tothe first storage node. A third step includes determining status ofsuccessfully processed messages of the number of sets of the pluralityof sets of messages by the first storage node. A fourth step includesdetermining the message processing status of the first storage nodebased on the status of sending messages and the status of successfullyprocessed messages.

The processing module determines the first message priority by a seriesof steps. A first step includes determining a number of sets of theplurality of sets of messages that involves the first storage node. Asecond step includes interpreting the system-level message processinginformation regarding the number of sets that involve the first storagenode to produce interpreted system-level message processing information.A third step includes interpreting the message processing status of thefirst storage node regarding the number of sets that involve the firststorage node to produce interpreted message processing status. A fourthstep includes applying a load balancing function in accordance theinterpreted system-level message processing information and theinterpreted message processing status to produce the first messagepriority.

For a second message of the set of messages, the method continues atstep 156 where the processing module determines a second messagepriority based on the system-level message processing information andmessage processing status of a second storage node. The method continuesat step 158 where the processing module sends the first message of theset of messages to the first storage node in accordance with the firstmessage priority. The method continues at step 160 where the processingmodule sends the second message of the set of messages to the secondstorage node in accordance with the second message priority.

The method continues at step 162 where the processing module updates thefirst message priority based on status of processing the set of messagesby a variety of approaches. A first approach includes, when the set ofmessages includes a set of write messages to write the set of encodeddata slices to the DSN, the processing module determines when a writethreshold number of the set of write messages have been sent to thedispersed storage network. When the write threshold number of the set ofwrite messages have been sent and the first message has not yet beensent to the first storage node, the processing module updates the firstmessage priority by reducing the first message priority. A secondapproach includes the processing module determining when a writethreshold number of the set of write messages have been successfullyprocessed by the dispersed storage network. When the write thresholdnumber of the set of write messages have been successfully processed andthe first message has not yet been sent to the first storage node, theprocessing module updates the first message priority by further reducingthe first message priority.

A third approach includes, when the set of messages includes a set ofread messages to read the set of encoded data slices from the DSN, theprocessing module determines when a decode threshold number of the setof read messages have been sent to the DSN. When the decode thresholdnumber of the set of read messages have been sent and the first messagehas not yet been sent to the first storage node, the processing moduleupdates the first message priority by reducing the first messagepriority. A fourth approach includes the processing module determiningwhen a decode threshold number of the set of read messages have beensuccessfully processed by the dispersed storage network. When the decodethreshold number of the set of read messages have been successfullyprocessed and the first message has not yet been sent to the firststorage node, the processing module updates the first message priorityby further reducing the first message priority.

A fifth approach includes, when the set of messages includes a set ofwrite messages to write the set of encoded data slices to the DSN, theprocessing module determines that a write threshold number of the set ofwrite messages have not been sent to the dispersed storage networkwithin a given time frame. When the write threshold number of the set ofwrite messages have not been sent in the given time period and the firstmessage has not yet been sent to the first storage node within the giventime period, the processing module updates the first message priority byincreasing the first message priority. A sixth approach includes, whenthe set of messages includes the set of write messages to write the setof encoded data slices to the DSN, the processing module determines whena write threshold number of the set of write messages have not beensuccessfully processed by the dispersed storage network within a giventime period. When the write threshold number of the set of writemessages have not been successfully processed within the given timeperiod and the first message has not yet been sent to the first storagenode, the processing module updates the first message priority byincreasing the first message priority.

A seventh approach includes, when the set of messages includes the setof read messages to read the set of encoded data slices from the DSN,the processing module determines when a decode threshold number of theset of read messages have not been sent to a dispersed storage networkwithin a given time period. When the decode threshold number of the setof read messages have not been sent within the given time period and thefirst message has not yet been sent to the first storage node in thegiven time period, the processing module updates the first messagepriority by increasing the first message priority.

An eighth approach includes, when the set of messages including the setof read messages to read the set of encoded data slices from the DSN,the processing module determines when a decode threshold number of theset of read messages have not been successfully processed by thedispersed storage network within a given time period. When the decodethreshold number of the set of read messages have not been successfullyprocessed within the given time period and the first message has not yetbeen sent to the first storage node, the processing module updates thefirst message priority by increasing the first message priority. Themethod continues at step 164 where the processing module updates thesecond message priority based on the status of processing the set ofmessages.

FIG. 7 is a flowchart illustrating an example of modifying a writesequence. The method begins at step 166 where a processing module (e.g.,of a dispersed storage (DS) processing module) receives data for storagein a dispersed storage network (DSN) memory. The receiving may includereceiving one or more of a data object, a data file, a data stream(e.g., video, audio), a filename, an object identifier (ID), a data ID,a block ID, a source name, a vault source name, a segment allocationtable vault source name, a user ID, a DSN resource ID, a vault ID, a DSunit ID, a DS unit storage set ID, a data size indicator, and a storagerequest. The method continues at step 168 where the processing moduleencodes the data using a dispersed storage error coding function toproduce a plurality of sets of encoded data slices in accordance withdispersal parameters. The dispersal parameters may include one or moreof a pillar width, a write threshold, a read threshold, a decodethreshold, an error coding matrix, an information dispersal algorithm(IDA) ID, an encryption algorithm ID, and an encryption key.

The method continues at step 170 where the processing module identifiesa set of DS units. The identifying may be based on one or more of avault look up based on a user ID, a source name generated based on thedata, an available to DS unit list, and a source name to DS unit IDtable lookup. The method continues at step 172 where the processingmodule obtains performance level information for the set of DS units.The performance level information includes one or more of averagelatency for message transmission, average latency for receiving amessage, queue resource utilization, number of messages in a queue(e.g., queue depth), write bandwidth utilization, and read bandwidthutilization. The obtaining may be based on one or more of a performancelevel historical record lookup, initiating a test, a query, andreceiving the performance level information.

The method continues at step 174 where the processing module determineswhether modifying another write sequence affiliated with a DS unit ofthe set of DS units improves write performance of the data. Thedetermining may be based on one or more of the performance levelinformation for the set of DS units and write sequence information ofother active write sequences (e.g., a number of favorable writeresponses corresponding to each other active write sequence, a writethreshold associated with each other active write sequence). Forexample, the processing module determines that modifying another writesequence affiliated with a DS unit of the DS units improves writeperformance of the data when the other write sequence has received awrite threshold number of favorable write responses and there are one ormore pending write requests associated with the other write sequence inone or more queues.

The method branches to step 178 when the processing module determinesthat modifying another write sequence affiliated with the DS unit of theset of DS units does not improve write performance of the data. Themethod continues to step 176 when the processing module determines thatmodifying another write sequence affiliated with the DS unit of the setof DS units improves write performance of the data. The method continuesat step 176 where the processing module facilitates modifying the writesequence affiliated with the DS unit. The facilitating may be based onone or more of the performance level information, the write sequenceinformation associated with the DS unit, and a modification approach.The modification approach includes selecting a write sequence to deleteand/or reprioritize including at least one of selecting a write sequenceassociated with an oldest write sequence, selecting a write sequenceassociated with a lowest priority level, selecting a write sequence thathas a most number of favorable write responses, selecting a writesequence that has at least a write threshold number of favorable writeresponses, and selecting a write sequence that has a most number offavorable write responses above an associated write threshold. Themethod continues to step 178.

The method continues at step 178 where the processing module facilitatessending the plurality of encoded data slices to the set of DS units. Thefacilitating includes one or more of generating a plurality of encodeddata slice write requests that includes the plurality of sets of encodeddata slices, establishing a priority level (e.g., in accordance with themodification of the write sequence affiliated with the DS unit) for atleast one encoded data slice write request of the plurality of encodeddata slice write requests, and storing the plurality encoded data slicewrite requests in corresponding queues for transmission to the set of DSunits

FIG. 8 is a flowchart illustrating another example of modifying a writesequence. The method begins at step 180 where a processing module (e.g.,of a dispersed storage (DS) processing module) accesses write sequenceinformation. The write sequence information includes one or more of aqueue depth, a priority level of a pending request, age of a pendingrequest, number of favorable write responses received so far, and awrite threshold number. The accessing may be based on one or more ofretrieving a message queue, lookup, receiving a request, a query, and anerror message.

The method continues at step 182 where the processing module determineswhether to elevate a priority level of a write sequence. The prioritylevel of the write sequence may be utilized in determining atransmission order of two or more pending write request messages of acommon queue such that a write request associated with a higher prioritylevel is transmitted prior to a write request associated with a lowerpriority level. The determining may be based on of the access writesequence information. For example, the processing module determines toelevate the priority level of the write sequence when the write sequenceis associated with an oldest write sequence of a plurality of writesequences that has not received a write threshold number of favorablewrite responses. Each write sequence of the plurality of write sequencesmay be associated with a different write threshold number. As anotherexample, the processing module determines to elevate the priority levelof the write sequence when the write sequence is associated with ahighest priority write sequence of the plurality of write sequences thathas not received the write threshold number of favorable writeresponses.

The method branches to step 186 when the processing module determinesnot to elevate the priority level of the write sequence. The methodcontinues to step 184 when the processing module determines to elevatethe priority level of the write sequence. The method continues at step184 where the processing module elevates the priority level of the writesequence. The elevating of the priority level includes at least one ofmodifying a priority level indicator of an associated write request in amessage queue to include a higher priority level number and reorderingpending write requests in the queue such that highest priority requestswill be transmitted next. The method continues to step 186.

The method continues at step 186 where the processing module determineswhether to lower the priority level of the write sequence. Thedetermining may be based on write sequence information. For example, theprocessing module determines to lower the priority level of the writesequence when the write sequence is associated with a write sequence ofa plurality of write sequences that has received a write thresholdnumber of favorable write responses. As another example, the processingmodule determines to lower the priority level of the write sequence whenthe write sequence is associated with a highest priority write sequenceof the plurality of write sequences that has received the writethreshold number of favorable write responses.

The method branches to step 190 when the processing module determinesnot to lower the priority level of the write sequence. The methodcontinues to step 188 when the processing module determines to lower thepriority level of the write sequence. The method continues at step 188where the processing module lowers the priority level of the writesequence. The lowering of the priority level includes at least one ofmodifying a priority level indicator of an associated write request in amessage queue to include a lower priority level number and reorderingpending write requests in the queue such that highest priority requestswill be transmitted next. The method continues to step 190. The methodcontinues at step 190 where the method ends.

FIG. 9 is a flowchart illustrating another example of modifying a writesequence, which include similar steps to FIG. 7. The method begins withsteps 166-168 of FIG. 7 where a processing module (e.g., of a dispersedstorage (DS) processing module) receives data for storage in a dispersedstorage network (DSN) memory and encodes the data using a dispersedstorage error coding function in accordance with dispersal parameters toproduce a plurality of sets of encoded data slices.

The method continues at step 196 where the processing module determineswhether sufficient write sequence operational resources are available.The write sequence operational resources includes one or more of amemory device utilized to store a queue, a memory space allocationutilized to store the queue, and an amount of available memory forstorage of write requests. The determining may include on one or more ofobtaining a data size indicator associated with the data, obtaining acurrent utilization level with regards to the write sequence operationalresources, comparing the data size indicator with the currentutilization level, and indicating that sufficient write sequenceoperational resources are available when the comparison is favorable.For example, the processing module determines that sufficient writesequence operational resources are available when an available resourceindicator of the current utilization level is greater than the data sizeindicator.

The method branches to step 200 when the processing module determinesthat sufficient write sequence operational resources are not available.The method continues to step 178 of FIG. 7 when the processing moduledetermines that sufficient write sequence operational resources areavailable. The method continues at step 178 of FIG. 7 where theprocessing module facilitates sending the plurality of encoded dataslices to the set of DS units.

The method continues at step 200 where the processing module determineswhether an acceptable modification of the write sequence operationalresources can be implemented to provide sufficient resources to storethe data when the processing module determines that sufficient writesequence operational resources are not available. The determining may bebased on one or more of operational resource availability and writesequence information of other active write sequences (e.g., a number offavorable write responses corresponding to other active write sequences,a write threshold associated with the other active write sequences). Forexample, the processing module determines that an acceptablemodification of the write sequence operational resources can beimplemented to provide sufficient resources to store the data when thewrite sequence information of another active write sequence indicatesthat a write threshold number of favorable write responses have beenreceived and pending write requests utilizing the write sequenceoperational resources may be deleted freeing up resources greater than anumber of resources required to store the data (e.g., the pending writerequests include one or more encoded data slices that are greater insize than one or more data slices of the data).

The method branches to step 204 when the processing module determinesthat an acceptable modification of the write sequence operationalresources can be implemented to provide sufficient resources to storethe data. The method continues to step 202 when the processing moduledetermines that an acceptable modification of the write sequenceoperational resources cannot be implemented to provide sufficientresources to store the data. The method continues at step 202 where theprocessing module facilitates responding to a resource issue. Thefacilitating includes one or more of sending a store data response thatincludes an indication of the resource issue, sending an error message,deactivating a process that receives new data storage requests, andactivating additional operational resources.

The method continues at step 204 where the processing module identifiesother write requests for deletion when the processing module determinesthat an acceptable modification of the write sequence operationalresources can be implemented to provide sufficient resources to storethe data. The identifying includes identifying at least one pendingwrite request associated with at least one other write sequence, whereinthe pending write request is associated with write sequence operationalresources greater than a number of write sequence operational resourcesrequired to store the data, and wherein the other write sequence hasreceived at least a write threshold number of favorable write responses.For example, the processing module identifies a pending write requestthat includes the largest encoded data slice of a plurality of encodeddata slices associated with a plurality of pending write requests,wherein each pending write request of the plurality of pending writerequests is associated with a write sequence that has received at leasta write threshold number of favorable write responses.

The method continues at step 206 where the processing module deletes theother write requests. For example, the processing module deletes theother write requests from one or more associated queues of the writesequence operational resources. The method continues with step 178 ofFIG. 7 where the processing module facilitates sending the plurality ofencoded data slices to a set of DS units.

FIG. 10 is a flowchart illustrating an example of retrieving data from adispersed storage network. The method begins at step 210 where aprocessing module (e.g., of a dispersed storage (DS) processing module)receives a data retrieval request for data stored as a plurality of setsof encoded data slices in a set of DS units of a dispersed storagenetwork (DSN) memory. The request may include receiving one or more of afilename, an object identifier (ID), a data ID, a block ID, a sourcename, a vault source name, a segment allocation table vault source name,a user ID, a DSN resource ID, a vault ID, a DS unit ID, a DS unitstorage set ID, a data size indicator, and a retrieval request.

The method continues at step 12 where the processing module selects aread threshold number of DS units of the set of DS units. The selectingmay be based on one or more of a vault look up based on a user ID, asource name of the data, an available DS unit list, a DS unitperformance history, a DS unit activity level in the gator, a DS unitestimated error rate, a number of pending requests associated with a DSunit, and a source name to DS unit ID table lookup. For example, theprocessing module selects the read threshold number of DS units toinclude DS units associated with a lowest number of pending requests ofan associated queue.

The method continues at step 214 where the processing module facilitatessending read requests to the read threshold number of DS units. Thefacilitating includes generating one or more sets of slice namescorresponding to the data retrieval request, generating one or more setsof encoded data slice requests that includes the one or more sets ofslice names, and storing the one or more sets of encoded data slicerequests in one or more corresponding queues for transmission to theread threshold number of DS units.

The method continues at step 216 where the processing module receives adecode threshold number of favorable read responses from a decodethreshold number of DS units of the read threshold number of DS units.For example, the processing module receives a first decode thresholdnumber of received encoded data slices that includes encoded data slicesof a common revision number, and wherein the received encoded dataslices each produce a favorable integrity test result (e.g., a receivedintegrity value compares favorably to a calculated integrity valueand/or the first decode threshold number of received encoded data slicesare dispersed storage error decoded to produce a data segment associatedwith a favorable integrity test result).

The method continues at step 218 where the processing module facilitatescanceling any pending read requests associated with other DS units ofthe read threshold number of DS units. For example, the processingmodule deletes associated read encoded data slice requests from one ormore message queues associated with other DS units of the read thresholdnumber of DS units (e.g., DS units not included in the decode thresholdnumber of DS units).

The method continues at step 220 where the processing module identifiesany non-responding DS units of the other DS units to produce identifiedDS units. For example, the processing module verifies DS units that didnot respond favorably to a corresponding read encoded data slicerequest. The method continues at step 222 where the processing modulefacilitates sending the identified DS units a read cancellation request.The facilitating includes generating one or more read cancellationrequests, storing the one or more cancellation requests in one or moreassociated message queues of the identified DS units, and prioritizingthe read cancellation requests (e.g., for immediate transmission to theidentified DS units). As such, a DS unit receiving a cancellationmessage cancels sending a read encoded data slice response in responseto an associated read encoded data slice request (e.g., does not startto send a response, stop sending a response in progress).

FIG. 11 is a diagram illustrating a directory and segment allocationtable structure. The structure includes a directory 224 and one or moresegment allocation tables (SAT) SAT A-B. The directory 224 includes afile name field 226 and a SAT source name field 228. The filename field226 includes one or more filename entries, wherein a filename includesat least one of a filename associated with data stored in a dispersedstorage network (DSN) memory, a filename associated with a file, anobject name, and a block number. The SAT source name field 228 includesone or more SAT source name entries corresponding to the one or morefilename entries. A SAT source name entry includes a vault source nameof a corresponding SAT stored as a plurality of SAT slices stored in theDSN memory at a DSN address of the SAT source name. For example, SATassociated with storing a movie file with a filename of movie.mp4 isstored at a SAT source name address of F46B and a SAT associated withstoring a movie file with a filename of trailer.mp4 is stored at a SATsource name address of 3D39.

A SAT of the one or more SATs includes a data region field 230, a startvault source name field 232, and a segment information field 234. Thedata region field 230 includes one or more data region entries, whereineach data region entry of the one more data region entries is associatedwith one or more data segments of corresponding data. The start vaultsource name field 232 includes one or more start vault source nameentries corresponding to the one or more data region entries. Each startvault source name entry includes a vault source name corresponding toDSN storage address of a first data segment of the one or more datasegments associated with a corresponding data region. The segmentinformation field 234 includes one or more segment information entriescorresponding to the one or more data region entries. Each segmentinformation entry includes one or more of a total length of the dataregion, a segment size within the data region, a segmentation approach(e.g., fixed, varying) utilized to segment the data into the one or moredata segments of the region, and a number of data segments correspondingto the data region.

The SAT may be utilized to generate a plurality of sets of slice namescorresponding to storage locations within the DSN memory of a pluralityof sets of encoded data slices generated from a corresponding dataregion of the data. For example, a plurality of sets of slice namescorresponding to a first data region of the movie file with the filenamemovie.mp4 may be generated from vault source names AA00-AA05 when acorresponding start vault source name is AA00 and corresponding segmentinformation indicates that six data segments are included within thefirst data region. As another example, a plurality of sets of slicenames corresponding to a second data region of the movie file with thefilename movie.mp4 may be generated from vault source names AA15-AA18when a corresponding start vault source name is AA15 and correspondingsegment information indicates that four data segments are includedwithin the second data region. As yet another example, a plurality ofsets of slice names corresponding to a second data region of the moviefile with the filename trailers.mp4 may be generated from vault sourcenames 687F-6885 when a corresponding start vault source name is 687F andcorresponding segment information indicates that seven data segments areincluded within the second data region.

FIG. 12A is a diagram illustrating another directory and segmentallocation table structure. The structure includes a directory 224 andone or more segment allocation tables (SAT) SAT A-C. The directory 224includes a file name field 226 and a SAT source name field 228. Each SATof the one or more SATs includes a data region field 230, a start vaultsource name field 232, and a segment information field 234. Thedirectory 224 and one or more SATs may be utilized to access one or morefiles stored within a dispersed storage network (DSN) memory. Thedirectory 224 and an additional SAT may be utilized to access a stitchedfile, wherein the stitched file includes data of two or more filesstored in the DSN memory.

For example, a file with a file name of movie.mp4 is stored in the DSNmemory as three data regions associated with start vault source namesAA00, AA15, and BB40 and a file with a file name of trailer.mp4 isstored in the DSN memory as two data regions associated with start vaultsource names D670 and 687F. A stitched file with a filename of both.mp4is accessed from the DSN memory as the five data regions associated withstart vault source names AA00, AA15, BB40, D670 and 687F. A new SAT C isgenerated that includes the five start vault source names andcorresponding segment information. The new SAT C corresponding to thestitched file both.mp4 is stored as encoded SAT slices at a SAT sourcename of 5FA7. The directory 224 is updated to include the filename ofboth.mp4 and the associated SAT source name of 5FA7. The method togenerate the new SAT corresponding to the stitched file is discussed ingreater detail with reference to FIG. 12B.

FIG. 12B is a flowchart illustrating an example of stitching data storedin a dispersed storage network (DSN) memory. The method begins at step236 where a processing module (e.g., of a dispersed storage (DS)processing module) determines to append a first data object and a seconddata object stored as encoded data slices in the DSN memory. Thedetermining may be based on one or more of user input, a request, acomparison of data object metadata, detecting a multi-part upload,receiving a resume function, receiving an append operation request,receiving a request to combine audio frames, a first data objectidentifier (ID), a second data object ID, and receiving a request tocombine video frames.

The method continues at step 238 where the processing module retrieves afirst segment allocation table (SAT) for the first data object. Forexample, the processing module accesses a DSN directory based on a filename of the first data object, extracts a SAT source name address fromthe directory, retrieves a plurality of encoded SAT slices from the DSNmemory utilizing the SAT source name address, and decodes the pluralityof encoded SAT slices to reproduce the first SAT. The method continuesat step 240 where the processing module retrieves a second SAT for thesecond data object.

The method continues at step 242 where the processing module generatesan appended SAT based on the first SAT and the second SAT. Thegenerating includes one or more of ordering data region information(e.g., a start vault source name and segment information associated witheach data region) associated with the first data object ahead of dataregion information associated with the second data object to producedata region information of the appended SAT, ordering the data regioninformation associated with the second data object ahead of the dataregion information associated with the first data object to produce thedata region information of the appended SAT, and deleting one or both ofthe first SAT and the second SAT (e.g., deleting encoded SAT slicesassociated with the corresponding SAT).

The method continues at step 244 where the processing module facilitatesstoring the appended SAT. The facilitating includes one or more ofdispersed storage error encoding the appended SAT to produce a pluralityof encoded appended SAT slices, obtaining an appended SAT source name(e.g., generating the appended SAT source name based on an appended filename, generating a plurality of slice names corresponding to theplurality of encoded appended SAT slices based on appended SAT sourcename, generating a plurality of write slice requests that includes theplurality of encoded appended SAT slices and the plurality of slicenames, and sending the plurality of write slice requests to the DSNmemory.

The method continues at step 246 where the processing module updates theDSN directory. The update includes one or more of adding a new entry anddeleting one or more entries associated with the first data object andthe second data object. The adding the new entry includes adding theappended file name and associated appended SAT source name to the DSNdirectory to produce an updated DSN directory and storing the updatedDSN directory in the DSN memory.

FIG. 13A is a diagram illustrating another directory and segmentallocation table structure. The structure includes a directory 224 andone or more segment allocation tables (SAT) SAT A-C. The directory 224includes a file name field 226 and a SAT source name field 228. Each SATof the one or more SATs includes a data region field 230, a start vaultsource name field 232, and a segment information field 234. Thedirectory 224 and one or more SATs may be utilized to access one or morefiles stored within a dispersed storage network (DSN) memory. Inaddition, the directory 224 and an additional SAT may be utilized toaccess a split file, wherein the split file includes one or more partialdata regions of one or more files stored in the DSN memory.

For example, a file with a file name of movie.mp4 is stored in the DSNmemory as three data regions associated with start vault source namesAA00, AA15, and BB40 and a file with a file name of trailer.mp4 isstored in the DSN memory as two data regions associated with start vaultsource names D670 and 687F. A split file with a filename of sampler.mp4is accessed from the DSN memory as three data regions associated withstart vault source names AA15, BB45, and 687F. A data region of the oneor more data regions may align with an entire data region of an existingdata region (e.g., where a start vault source name of a data region ofthe one or more data regions is the same as a start vault source name ofthe existing data region). Alternatively, a data region of the one ormore data regions may align with a portion of an existing data region(e.g., where a start vault source name of a data region of the one ormore data regions is offset from the start vault source name of theexisting data region). Segment information associated with a data regionof the one or more data regions may align with corresponding segmentinformation of the existing data region. Alternatively, the segmentinformation associated with the data region may include differentsegment information (e.g., a fewer number of data segments as comparedto the corresponding segment information of the existing data region).

A new SAT C is generated that includes the three start vault sourcenames and corresponding segment information. The new SAT C correspondingto the split file sampler.mp4 is stored as encoded SAT slices at a SATsource name address of 38AB. The directory is updated to include thefile name of sampler.mp4 and the associated SAT source name of 38AB. Themethod to generate the new SAT corresponding to the split file isdiscussed in greater detail with reference to FIGS. 13B-13D.

FIG. 13B is a flowchart illustrating an example of splitting data storedin a dispersed storage network (DSN) memory. The method begins at step248 where a processing module (e.g., of a dispersed storage (DS)processing module) determines to split out one or more portions of oneor more data objects stored as encoded data slices in the DSN memory.The determining may be based on one or more of user input, a request, acomparison of data object metadata, detecting a multi-part upload,receiving a resume function, receiving a split operation request,receiving a request to combine audio frames, a first data objectidentifier (ID), a second data object ID, and receiving a request tocombine video frames.

The method continues at step 250 where the processing module retrievesone or more segment allocation tables (SATs) for one or more dataobjects. For example, the processing module accesses a DSN directorybased on a file name of a first data object, extracts a SAT source nameaddress from the directory, retrieves a plurality of encoded SAT slicesfrom the DSN memory utilizing the SAT source name address, and decodesthe plurality of encoded SAT slices to reproduce a first SAT.

The method continues at step 252 where the processing module identifiesone or more data sections corresponding to desired data. The identifyingmay be based on one or more of a user input, a request, lookup, a list,a priority level indicator associated with the one or more datasections, a copyright indicator, and a request. For example, theprocessing module identifies a first data section that includes a 30second section of a music file when a priority level associated with the30 second section indicates a priority level greater than a prioritythreshold. As another example, the processing module identifies a seconddata section that includes a 60 second movie scene in accordance with asplit list.

The method continues at step 254 where the processing module generates anew SAT based on the one or more data sections and the one or more SATs.The generating includes one or more of generating data regioninformation for each of the one or more data sections (e.g., a startvault source name and segment information associated with each dataregion), ordering the data region information to produce the new SAT inaccordance with one or more of an identification order, a user input, arequest, and a list; and deleting at least one of the one or more SATs(e.g., deleting encoded SAT slices associated with the at least oneSAT).

The method continues at step 256 where the processing module facilitatesstoring the new SAT. The facilitating includes one or more of dispersedstorage error encoding the new SAT to produce a plurality of encoded newSAT slices, obtaining a new SAT source name (e.g., generating the newSAT source name based on a new file name), generating a plurality ofslice names corresponding to the plurality of encoded new SAT slicesbased on the new SAT source name, generating a plurality of write slicerequests that includes the plurality of encoded new SAT slices and theplurality of slice names, and sending the plurality of write slicerequests to the DSN memory.

The method continues at step 258 where the processing module updates theDSN directory. The updating includes one or more of adding a new entryand deleting one or more entries associated with the one or more dataobjects. The adding the new entry includes adding the new file name andassociated new SAT source name to the DSN directory to produce anupdated DSN directory and storing the updated DSN directory in the DSNmemory.

FIG. 13C is a schematic block diagram of another embodiment of acomputing system that includes a computing device 260 and a dispersedstorage network (DSN) memory 22 of a dispersed storage network. Thecomputing device 260 includes a dispersed storage (DS) processing 262.The computing device 260 may be implemented as at least one of a userdevice, a DS processing unit, and a DS unit. The DS processing 262includes a receive request module 264, an identify module 266, anaddress module 268, and a generate segment allocation table (SAT) module270. The system functions to access the DSN memory 22 with regards tocreating a new file from portions of a set of existing files 272 (e.g.,one or more existing files). An existing file of the set of existingfiles 272 includes a plurality of data segments that is stored as aplurality of sets of encoded data slices in the DSN memory 22. Theexisting file of the set of existing files 272 may include a set of dataregions. A data region of the set of data regions includes a consecutivesubset of the plurality of data segments.

The receive request module 264 receives a file creation request 274 tocreate the new file from the portions of the set of existing files 272.The identify module 266 identifies data segments from the plurality ofdata segments of the set of existing files 272 corresponding to theportions of the set of existing files 272 to produce identified datasegments. The identify module 266 identifies data segments by at leastone of a variety of approaches. A first approach includes receiving,within the file creation request 274, identification of the identifieddata segments. A second approach includes searching, based on a searchcriteria (e.g., from the file creation request 274), the set of existingfiles 272, or addressing information thereof, to identify the datasegments.

The identify module 266 identifies sets of encoded data slices (e.g.,one set per data segment) of the plurality of sets of encoded dataslices of the identified data segments to produce identified sets ofencoded data slices 276. When the existing file includes the set of dataregions, the identify module 266 identifies the sets of encoded dataslices 276 by retrieving a set of segment allocation tables 278 for theset of existing files 272 based on the file creation request 274. Asegment allocation table of the set of segment allocation tablesincludes a plurality of entries. An entry of the plurality of entriesincludes a DSN address (e.g., a source name) regarding storage of a datasegment of the plurality of data segments in the DSN memory. Forexample, the identify module 266 retrieves the set of segment allocationtables 278 from the DSN memory 22 (e.g., stored as sets of encoded SATslices).

When the existing file includes the set of data regions, the identifymodule 266 identifies data regions from the set of data regions of theset of existing files 272 corresponding to the portions of the set ofexisting files to produce identified data regions. Next, the identifymodule 266 identifies the sets of encoded data slices of the pluralityof sets of encoded data slices of the identified data regions to producethe identified sets of encoded data slices 276. The identify module 266identifies the sets of encoded data slices 276 by retrieving the set ofsegment allocation tables 278 for the set of existing files based on thefile creation request. An entry of the plurality of entries of thesegment allocation table of the set of segment allocation tablesincludes a DSN address regarding storage of a data region of the set ofdata segments of the set of existing files in the DSN memory andsegmentation information regarding the dividing of the data region intothe consecutive subset of the plurality of data segments.

The address module 268 determines addressing information 280 for theidentified sets of encoded data slices 276 from existing addressinginformation of the set of existing files 272. When the existing fileincludes the set of data regions, the address module 268 determines theaddressing information 280 for the identified sets of encoded dataslices 276 from the set of segment allocation tables 278 associated withthe set of existing files 272. The set of segment allocation tables 278includes the existing addressing information of the set of existingfiles 272 at data region level. As such, data segments are grouped intoa data region and the addressing information is for the region andaddressing information for individual sets of slices may be determinedfrom the region addressing information. Addressing information for anidentified set of encoded data slices of the identified sets of encodeddata slices 276 includes a start segment vault source name (e.g.,including vault identifier, a generation identifier, an object numbercorresponding to the existing file, and a segment number) correspondingto a data region associated with the identified set of encoded dataslices and segmentation information. The segmentation informationincludes one or more of a segment size of data segments of the dataregion, a total length of the data region (e.g., number of bytes), and asegmentation approach of the data region (e.g., fixed segment sizes,variable segment sizes).

The address module 268 identifies a vault source name of an identifiedset of encoded data slices for a first data segment of a consecutivesubset of data segments of the identified data segments corresponding tothe data region to produce the start segment vault source name. Theaddress module 268 extracts a segment size from the existing addressinginformation of the data region to produce the segment size. The addressmodule 268 extracts a segmentation approach from the existing addressinginformation of the data region to produce the segmentation approach. Theaddress module 268 determines a number of bytes (e.g., number of datasegments multiplied by the segment size) of the consecutive subset ofdata segments from the existing addressing information of the dataregion to produce the total length of the data region.

The generate segment allocation table module 270 generates a segmentallocation table 282 for the new file based on the addressinginformation 280 of the identified sets of encoded data slices 276 suchthat the new file is created without duplication of the portions of theexisting files. The generate SAT module 270 generates the segmentallocation table 282 for the new file by a series of steps. A first stepincludes identifying consecutive data segments of the identified datasegments. A second step includes grouping the consecutive data segmentsinto a region. A third step includes generating addressing informationfor the region which is stored as an entry in the segment allocationtable. Individual addressing information of the identified sets ofencoded data slices 276 is determinable from the addressing informationof the region.

The generate SAT module 270 stores the segment allocation table 282 inat least one of a local memory and the DSN memory 22. The generate SATmodule 270 stores the segment allocation table 282 in the DSN memory 22using a series of steps. In a first step, the generate SAT module 270obtains a segment allocation table vault source name for the segmentallocation table 282 (e.g., generate a new vault source name). In asecond step, the generate SAT module 270 encodes the segment allocationtable 282 using a dispersed storage error coding function to produceencoded table slices 284. In a third step, the generate SAT module 270outputs the encoded table slices 284 to the DSN memory 22 for storagetherein utilizing the segment allocation table vault source name.

FIG. 13D is a flowchart illustrating an example of creating a new file.The method begins at step 290 where a processing module (e.g., of adispersed storage (DS) processing unit) receives a file creation requestto create a new file from portions of a set of existing files. Anexisting file of the set of existing files includes a plurality of datasegments that is stored as a plurality of sets of encoded data slices ina dispersed storage network (DSN) memory. The existing file of the setof existing files may include a set of data regions. A data region ofthe set of data regions includes a consecutive subset of the pluralityof data segments.

The method continues at step 292 where the processing module identifiesdata segments from the plurality of data segments of the set of existingfiles corresponding to the portions of the set of existing files toproduce identified data segments. The identifying data segments includesat least one of the variety of approaches. A first approach includesreceiving, within the file creation request, identification of theidentified data segments. A second approach includes searching, based ona search criteria, the set of existing files, or addressing informationthereof, to identify the data segments.

The method continues at step 294 where the processing module identifiessets of encoded data slices (e.g., one set per data segment) of theplurality of sets of encoded data slices of the identified data segmentsto produce identified sets of encoded data slices. When the existingfile includes the set of data regions, the identifying the sets ofencoded data slices includes retrieving a set of segment allocationtables for the set of existing files based on the file creation request.A segment allocation table of the set of segment allocation tablesincludes a plurality of entries and an entry of the plurality of entriesincludes a DSN address regarding storage of a data segment of theplurality of data segments in the DSN memory.

Alternatively, when the existing file includes the set of data regions,the processing module identifies data regions from the set of dataregions of the set of existing files corresponding to the portions ofthe set of existing files to produce identified data regions. Next, theprocessing module identifies the sets of encoded data slices of theplurality of sets of encoded data slices of the identified data regionsto produce the identified sets of encoded data slices. The identifyingthe sets of encoded data slices includes retrieving the set of segmentallocation tables for the set of existing files based on the filecreation request (e.g., a segment allocation table of the set of segmentallocation tables includes a plurality of entries). An entry of theplurality of entries includes a DSN address regarding storage of a dataregion of the set of data segments of the set of existing files in theDSN memory and segmentation information regarding the dividing of thedata region into the consecutive subset of the plurality of datasegments.

The method continues at step 296 or the processing module determinesaddressing information for the identified sets of encoded data slicesfrom existing addressing information of the set of existing files. Whenthe existing file includes the set of data regions, processing moduledetermines the addressing information for the identified sets of encodeddata slices from a set of segment allocation tables associated with theset of existing files. The set of segment allocation tables includes theexisting addressing information of the set of existing files at dataregion level. The addressing information for the identified set ofencoded data slices of the identified sets of encoded data slicesincludes a start segment vault source name corresponding to a dataregion associated with the identified set of encoded data slices andsegmentation information. The segmentation information includes one ormore of a segment size of data segments of the data region, a totallength of the data region, and a segmentation approach of the dataregion.

The method continues at step 298 where the processing module generates asegment allocation table for the new file based on the addressinginformation of the identified sets of encoded data slices such that thenew file is created without duplication of the portions of the existingfiles. The generating the segment allocation table for the new fileincludes a series of steps. A first step includes identifyingconsecutive data segments of the identified data segments. A second stepincludes grouping the consecutive data segments into a region. A thirdstep includes generating addressing information for the region, which isstored as an entry in the segment allocation table, wherein individualaddressing information of the identified sets of encoded data slices isdeterminable from the addressing information of the region.

The method continues at step 300 where the processing module obtains asegment allocation table vault source name for the segment allocationtable. The obtaining includes at least one of receiving, retrieving, andgenerating. The generating includes generating an object number based ona random number generator and associating the object number with afilename of the new file (e.g., modifying a file directory to includethe file name in the segment allocation table vault source name). Themethod continues at step 302 where the processing module encodes thesegment allocation table using a dispersed storage error coding functionto produce encoded table slices. The method continues at step 304 wherethe processing module outputs the encoded table slices to the DSN memoryfor storage therein utilizing the segment allocation table vault sourcename.

FIG. 14A is a schematic block diagram of another embodiment of acomputing system that includes a dispersed storage (DS) processing unit16 and two or more dispersed storage (DS) units 1-2. The DS processingunit 16 sends slices 11 to the two or more DS units 1-2 for storagetherein and retrieves slices 11 from the two or more DS units 1-2. EachDS unit of the two or more DS units 1-2 includes one or more memories.For example, DS unit 1 includes memories 1-4 and DS unit 2 includesmemories 1-3.

Each memory of the one or more memories may be associated with acorresponding dispersed storage network (DSN) address range. Forexample, memory 1 of DS unit 1 is associated with DSN address range 1,memory 2 of DS unit 1 is associated with DSN address range 2, memory 3of DS unit 1 is associated with DSN address range 3, memory 4 of DS unit1 is associated with DSN address range 4, memory 1 of DS unit 2 isassociated with DSN address range 5, memory 2 of DS unit 2 is associatedwith DSN address range 6, and memory 3 of DS unit 2 is associated withDSN address range 7.

In an example of operation, the DS processing unit 16 sends a sliceretrieval request to DS unit 1, wherein the request includes a slicename of address range 3. DS unit 1 retrieves and encoded data sliceutilizing the slice name from memory 3 and sends the encoded data sliceto the DS processing unit 16.

FIG. 14B is a schematic block diagram of another embodiment of acomputing system that includes two or more dispersed storage (DS) units1-2. Each DS unit of the two or more DS units 1-2 includes one or morememories for storing slices based on dispersed storage network (DSN)address range assignments of the one or more memories. For example, DSunit 1 includes memories 1-4 utilized for storing slices associated withDSN address range assignments for memories 1-4 of DS unit 1 and DS unit2 includes memories 1-3 utilized for storing slices associated with DSNaddress range assignments for memories 1-3 of DS unit 2.

A memory of the one or more memories may fail from time to time. Assuch, all, some, or none of slices stored in the memory may beaccessible. A DS unit may transfer at least some slices stored in anassociated failing memory to one or more other memories when the memoryis detected as the failing memory. The transferring may include one ormore of retrieving the at least some of the slices stored in the failingmemory, identifying the one or more other memories, storing the at leastsome of the slices in the one or more other memories, and updating DSNaddress range assignments for the memory and the one or more othermemories. For example, DS unit 1 retrieves at least some slices storedin a failing memory 3, identifies memories 2 and 4 as the one or moreother memories, sends transfer slices A of the at least some of theslices to memory 2, sends transfer slices B of the at least some of theslices to memory 4, updates a DSN address range assignment for memory 3(e.g., no assignment), updates a DSN address range assignment for memory2 to include an address range associated with transfer slices A, andupdates a DSN address range assignment for memory 4 to include anaddress range associated with transfer slices B. The method to detect afailing memory and transfer slices is discussed in greater detail withreference to FIGS. 15-17B.

FIG. 14C is a schematic block diagram of another embodiment of acomputing system that includes two or more dispersed storage (DS) units1-2. Each DS unit of the two or more DS units 1-2 includes one or morememories for storing slices based on dispersed storage network (DSN)address range assignments of the one or more memories. For example, DSunit 1 includes memories 1-4 utilized for storing slices associated withDSN address range assignments for memories 1-4 of DS unit 1 and DS unit2 includes memories 1-3 utilized for storing slices associated with DSNaddress range assignments for memories 1-3 of DS unit 2.

A memory on the one or more memories may fail from time to time. Assuch, all, some, or none of slices stored in the memory may beaccessible. A DS unit may transfer at least some slices stored in anassociated failing memory to one or more other memories when the memoryis detected as the failing memory. The transferring may include one ormore of retrieving the at least some of the slices stored in the failingmemory, identifying the one or more other memories, identifying anotherDS unit, storing the at least some of the slices in the one or moreother memories, storing at least some of the slices in the other DSunit, and updating DSN address range assignments for one or more of thememory, the one or more other memories, and the other DS unit. Forexample, DS unit 1 retrieves at least some slices stored in a failingmemory 4, identifies memory 3 of DS unit 1 as the one or more othermemories, identifies DS unit 2 as another DS unit, sends transfer slicesA of the at least some of the slices to memory 3, sends transfer slicesB of the at least some of the slices to DS unit 2, updates a DSN addressrange assignment for memory 4 (e.g., no assignment), updates a DSNaddress range assignment for memory 3 to include an address rangeassociated with transfer slices A, and updates a DSN address rangeassignment for DS unit 2 to include an address range associated withtransfer slices B. DS unit 2 may update a DSN address range assignmentfor memory 1 of DS unit 2 to include the address range associated withtransfer slices B. The method to detect a failing memory and transferslices is discussed in greater detail with reference to FIGS. 15-17B.

FIG. 15 is a flowchart illustrating an example of detecting a failingmemory. The method begins at step 306 where a processing module (e.g.,of a dispersed storage (DS) module, of a DS unit) detects a failingmemory. The detecting includes detecting at least one of a memorycontroller error, a read error (e.g., a missing slice, a corrupt slice),a write error, unfavorable access latency, an error message, a testerror, an error correlation to a common memory (e.g., memories 2 and 3slow down every time memory 1 is accessed), an unfavorable memory age,an unfavorable memory usage history, an unfavorable memory test result,and an unfavorable slice rebuilding history.

The method continues at step 308 where the processing module determineswhether the failing memory has failed an unfavorable number of times.The determination includes comparing a number of cumulative failures toa number of failures threshold and indicating that the failing memoryhas failed an unfavorable number of times when the comparison isunfavorable. For example, the processing module determines that thefailing memory has failed an unfavorable number of times when a numberof cumulative failures is four and a number of failures threshold isthree. The method branches to step three 318 when the processing moduledetermines that the failing memory has failed an unfavorable number oftimes. The method continues to step 310 when the processing moduledetermines that the failing memory has not failed an unfavorable numberof times.

The method continues at step 310 where the processing module temporarilyremoves the failing memory from service. The temporarily removing thefailing memory from service includes one or more of incrementing anumber of cumulative failures indicator by one, disabling access to thememory, and updating a dispersed storage network (DSN) address rangeassignment table to indicate that the failing memory is the longerassigned to a previously assigned DSN address range.

The method continues at step 312 where the processing module performs adiagnostic test on the failing memory to produce a diagnostic result.The performing the diagnostic test includes one or more of facilitatingresetting the failing memory, writing a test pattern to the failingmemory, reading the test pattern from the failing memory to produce arecovered test pattern, comparing the recovered test pattern to the testpattern, measuring an access latency, and generating the diagnosticresult based on the comparison and/or the access latency (e.g., indicatea failure when the comparison is unfavorable, indicate a failure whenthe access latency compares unfavorably to an access latency threshold).

The method continues at step 314 where the processing module determineswhether the diagnostic result is favorable. The determining includes atleast one of extracting a functional result, extracting a performanceresult, comparing the functional result to a functional threshold,comparing the performance result to a performance threshold, andindicating that the diagnostic result is favorable when functional andperformance comparisons are favorable. The method branches to step 318when the processing module determines that the diagnostic result isunfavorable. The method continues to step 316 when the processing moduledetermines that the diagnostic result is favorable. The method continuesat step 316 where the processing module reinstates the failing memoryback in service. The reinstating includes at least one of enablingaccess to the memory and updating the DSN address range assignment tableto indicate that the failing memory is assigned to the previouslyassigned DSN address range when the processing module previously updatedthe DSN address range assignment table to indicate that the failingmemory is the longer assigned to the previously assigned DSN addressrange.

The method continues at step 318 where the processing module selects atleast some encoded data slices stored in the failing memory for transferto produce transfer slices. The selecting includes selecting slices inaccordance with a slice selection scheme, wherein the scheme includes atleast one of selecting all slices, priority slices, slices accessed mostfrequently, slices indicated by predetermination, slices indicated basedon a query, and slices indicated in a received message.

The method continues at step 320 where the processing module selects oneor more other memories to store the transfer slices. The selecting maybe based on one or more of the DSN address range assignment table, amemory availability indicator, a memory performance level indicator, andidentifying the one or more other memories that include a DSN addressrange assignment that is adjacent to the priestly assigned DSN addressrange of the failing memory.

The method continues at step 322 where the processing module transfersthe transfer slices from the failing memory to the one or more othermemories. For example, the processing module retrieves the transferslices from the failing memory, stores the transfer slices in the one ormore other memories, and updates the DSN address range assignment tableto indicate that the previously assigned DSN address range is nowassigned to the one or more other memories. The method continues at step324 where the processing module permanently removes the failing memoryfrom service. The permanently removing includes disabling access to thefailing memory, generating an error message that includes a removal fromservice indicator and a diagnostic result, and sending the error message(e.g., to a DS managing unit).

FIG. 16 is a flowchart illustrating another example of detecting afailing memory, which includes similar steps to FIG. 15. The methodbegins with step 306 of FIG. 15 where a processing module (e.g., of adispersed storage (DS) module, of a DS unit) detects a failing memory.The method continues with steps 318, 320, and 322 of FIG. 15 where theprocessing module selects at least some encoded data slices stored inthe failing memory for transfer to produce transfer slices, selects oneor more other memories to receive the transfer slices, and transfers thetransfer slices from the failing memory to the one or more othermemories.

The method continues with steps 310, 312, and 314 of FIG. 15 where theprocessing module temporarily removes the failing memory from service,performs a diagnostic test of the failing memory to produce a diagnosticresult, and determines whether the diagnostic result is favorable. Themethod branches to step 316 of FIG. 15 when the processing moduledetermines that the diagnostic result is favorable. The method continuesto step 324 of FIG. 15 when the processing module determines that thediagnostic test result is not favorable. The method continues at step 3to 24 of FIG. 15 where the processing module permanently removes thefailing memory from service. The method continues at step 316 where theprocessing module reinstates the failing memory back in service when theprocessing module determines that the diagnostic result is favorable.

FIG. 17A is a flowchart illustrating an example of migrating slices. Themethod begins at step 326 where a processing module (e.g., of adispersed storage (DS) module, of a DS unit) identifies a memoryassociated with slices to migrate. The identifying includes one or moreof receiving a failing memory indicator, receiving a message, performinga memory test, interpreting a memory test result, and receiving arequest. The method continues at step 328 where the processing moduleobtains a dispersed storage network (DSN) address range of the memory.For example, the processing module accesses a local DSN address rangeassignment table utilizing an identifier of the memory to retrieve theDSN address range of the memory.

The method continues at step 330 where the processing module obtains aDSN address range of a DS unit associated with the memory. The obtainingincludes at least one of accessing the local DSN address rangeassignment table utilizing the DSN address range of the memory toretrieve the DSN address range of the DS unit associated with the memoryand accessing a global DSN address to physical location table toretrieve the DSN address range of the DS unit associated with thememory.

The method continues at step 332 where the processing module determineswhether the DSN address range of the memory is associated with an end ofthe DSN address range of the DS unit. The determination may be based onor more of comparing the DSN address range of the memory with the DSNaddress range of the DS unit, indicating that the DSN address range ofthe memory aligns with a lower end of the DSN address range of the DSunit when a starting DSN address of the DSN address range of the memoryis substantially the same as a starting DSN address of the DSN addressrange of the DS unit, and indicating that the DSN address range of thememory aligns with an upper end of the DSN address range of the DS unitwhen an ending DSN address of the DSN address range of the memory issubstantially the same as an ending DSN address of the DSN address rangeof the DS unit. The method branches to step 340 when the processingmodule determines that the DSN address range of the memory is associatedwith the end of the DSN address range of the DS unit. The methodcontinues to step 334 when the processing module determines that the DSNaddress range of the memory is not associated with the end of the DSNaddress range of the DS unit.

The method continues at step 334 where the processing module identifiesadjacent memories. The identifying includes one or more of accessing thelocal DSN address range assignment table, identifying a lower adjacentmemory of the adjacent memories, wherein the lower adjacent memory isassigned a DSN address range immediately below the DSN address range ofthe memory, and identifying a higher adjacent memory of the adjacentmemories, wherein the higher adjacent memory is assigned a DSN addressrange immediately above the DSN address range of the memory.

The method continues at step 336 where the processing module determinesa slice transfer approach. The slice transfer approach indicates whichslices of the slices to transfer to transfer to the lower adjacentmemory and which slices of the slices to transfer to transfer to thehigher adjacent memory. The determination may be based on one or more ofa memory availability indicator of the lower adjacent memory, a memoryavailability indicator of the higher adjacent memory, apredetermination, a lookup, a performance level of the lower adjacentmemory, a performance level of the higher adjacent memory, and amessage.

The method continues at step 338 where the processing module transfersthe slices to transfer from the memory to the adjacent memories inaccordance with the slice transfer approach. The transferring includesat least one of transferring slices to transfer from the memory to theadjacent memories and updating the DSN address range assignment table inaccordance with the slice transfer approach. For example, the processingmodule retrieves slices to transfer to the lower adjacent memory fromthe memory, stores the slices to transfer to the lower adjacent memoryin the lower adjacent memory, retrieves slices to transfer to the higheradjacent memory from the memory, stores the slices to transfer to thehigher adjacent memory in the higher adjacent memory, and updates thelocal DSN address range assignment table to indicate that the DSNaddress range of the memory is now affiliated with the adjacent memoriesin accordance with the slice transfer approach.

The method continues at step 340 where the processing module identifiesan adjacent memory of the DS unit when the processing module determinesthat the DSN address range of the memory is associated with the end ofthe DSN address range of the DS unit. The identifying includes one ormore of accessing the local DSN address range assignment table andidentifying an adjacent memory, wherein the adjacent memory is assigneda DSN address range immediately next to the DSN address range of thememory (e.g., lower or higher).

The method continues at step 342 where the processing module determineswhether to utilize an adjacent DS unit. The determination may be basedon one or more of a memory availability level of the adjacent memory, amemory availability level of the adjacent DS unit, a network activitylevel, and a network configuration indicator. For example, theprocessing module determines to utilize the adjacent DS unit when thenetwork configuration indicator indicates that the adjacent DS unit islocated at a common site as the DS unit. As another example, theprocessing module determines not to utilize the adjacent DS unit whenthe network configuration indicator indicates the adjacent DS unit islocated at a different site as a site of the DS unit.

The method branches to step 346 when the processing module determines toutilize the adjacent DS unit. The method continues to step 344 when theprocessing module determines not to utilize the adjacent DS unit. Themethod continues at step 344 where the processing module transfersslices from the memory to the adjacent memory. For example, theprocessing module retrieves all the slices to transfer from the memory,stores all the slices to transfer in the adjacent memory, and updatesthe local DSN address range assignment table to indicate that the DSNaddress range of the memory is now affiliated with the adjacent memory.

The method continues at step 346 where the processing module determinesa slice transfer approach utilizing the adjacent DS unit when theprocessing module determines to utilize the adjacent DS unit. The slicetransfer approach indicates which slices of the slices to transfer totransfer to the adjacent memory and which slices of the slices totransfer to transfer to the adjacent DS unit. The determining may bebased on one or more of the network activity level, the networkconfiguration indicator, a memory availability indicator of the adjacentmemory, a memory availability indicator of the adjacent DS unit, apredetermination, a lookup, a performance level of the adjacent memory,a performance level of the adjacent DS unit, and a message.

The method continues at step 348 where the processing module transfersthe slices to transfer from the memory to the adjacent memory and theadjacent DS unit in accordance with the slice transfer approach. Thetransferring includes at least one of transferring at least some of theslices to transfer from the memory to the adjacent memory, transferringat least some of the slices to transfer from the memory to the DS unit,updating the local DSN address range assignment table in accordance withthe slice transfer approach, and updating the global DSN address rangeassignment table in accordance with the slice transfer approach. Forexample, the processing module retrieves slices to transfer to theadjacent memory from the memory, stores the slices to transfer to theadjacent memory in the adjacent memory, retrieves slices to transfer tothe adjacent DS unit from the memory, stores the slices to transfer tothe adjacent DS unit in the adjacent DS unit, updates the local DSNaddress range assignment table to indicate that at least some the DSNaddress range of the memory is now affiliated with the adjacent memoryin accordance with the slice transfer approach, and updates the globalDSN address range assignment table to indicate that a remaining portionthe DSN address range of the memory is now affiliated with the adjacentDS unit in accordance with the slice transfer approach.

FIG. 17B is a diagram of an embodiment of a distributed storage network(DSN) address mapping that includes a global DSN address space 350illustrated as a circle including a first address 352 of the global DSNaddress space 350 to start the circle and ending with a last address 354of the global DSN address space 350. An address of the global DSNaddress space 350 may include a slice name associated with an encodeddata slice stored in a DSN memory. The slice name may include multiplefields. A most significant field may include an address sector field. Anumber of permutations of the address sector field corresponds to anumber a pillar width (e.g., number of pillars) of dispersal parametersutilized to generate the encoded data slices. For example, the pillarwidth is four when the number of permutations of the address sectorfield is four. The global DSN address space 350 is divided into aplurality of address sectors. For example, the plurality of addresssectors includes a first address sector 356, a second address sector358, a third address sector 360, and a fourth address sector three 362when four sector addresses are utilized.

Recovery reliability of retrieving data from the DSN memory is improvedby aligning storage units with the plurality of address sectors. Forexample, a first dispersed storage (DS) unit is utilized to storeencoded data slices that correspond to slice names that include a firstaddress sector field value, a second dispersed storage (DS) unit isutilized to store encoded data slices that correspond to slice namesthat include a first address sector field value, a third dispersedstorage (DS) unit is utilized to store encoded data slices thatcorrespond to slice names that include a third address sector fieldvalue, and a fourth dispersed storage (DS) unit is utilized to storeencoded data slices that correspond to slice names that include a fourthaddress sector field value. Recovery of data is possible by retrievingjust three encoded data slices from three DS units of the four DS unitswhen a decode threshold number of the dispersal parameters is three.

FIG. 17C is a diagram of an embodiment of a distributed storage network(DSN) address mapping that includes a portion of a global DSN addressspace 350 illustrated as a portion of a circle. The global DSN addressspace 350 is divided into a plurality of address sectors. For example,the plurality of address sectors includes a first address sector 356. Anaddress sector includes one or more local DSN address ranges. A localDSN address range is a portion of an address sector of the plurality ofaddress sectors. For example, the first address sector 356 includes afirst local DSN address range 364 and a second local DSN address range366. For each local DSN address range of the one or more local DSNaddress ranges, the local DSN address range is associated with adispersed storage (DS) unit utilized to store encoded data slices thatare assigned addresses within the local DSN address range. For example,a DS unit 1 is utilized to store encoded data slices that are assignedaddresses within the first local DSN address range 364 and a DS unit 2is utilized to store encoded data slices that are assigned addresseswithin the second local DSN address range 366.

Two or more DS units associated with a common address sector may beimplemented at a common site without impacting system reliability evenwhen the site is off-line since a decode threshold number of encodeddata slices may be available by retrieving the decode threshold numberof encoded data slices associated with a decode threshold number ofother address sectors (e.g., from a decode threshold number of othersites). From time to time encoded data slices stored in one or morememory devices of a DS unit may require transfer to another memorydevice of the DS units or to a memory device of another DS unitassociated with a common sector with the DS unit. For example, the onemore memory devices may fail. As another example, the one more memorydevices may reach and end-of-life time period. The DS unit may functionto identify the other DS unit and determine whether to transfer encodeddata slices to the other DS unit when requiring to transfer the encodeddata slices to another memory device. The method to migrate encoded dataslices is discussed in greater detail with reference to FIGS. 17D-E.

FIG. 17D is a schematic block diagram of another embodiment of acomputing system that includes a dispersed storage (DS) unit 370 andanother DS unit 36. The DS unit 370 includes a DS processing module 372and a plurality of memory devices 374. The DS unit 370 functions tostore encoded data slices 376. Data segments of data (e.g., one or moredata objects, one or more files, etc.) are encoded (e.g., by a DSprocessing unit) using a dispersed storage error coding function toproduce a plurality of sets of encoded data slices. The stored encodeddata slices 376 includes one or more encoded data slices of at leastsome sets of the plurality of sets of encoded data slices.

The DS unit 370 is assigned a local distributed storage network (DSN)address range. A global DSN address space is divided into a plurality ofaddress sectors. The local DSN address range is a portion of an addresssector of the plurality of address sectors. The DS unit 370 and theother DS unit 36 are assigned to the address sector by a systemmanagement function. The DS unit address range is not tied to thedispersed storage error coding function when the system managementfunction assigns DS unit 370 and the other DS unit 36 to the addresssector. The plurality of sets of encoded data slices have a plurality ofsets of slice names. A slice name of the plurality of sets of slicesnames includes an address sector field (e.g., pillar number) thatidentifies one of the plurality of address sectors. The address sectorfield corresponds to an ordering of the dispersed storage error codingfunction.

The DS processing module 372 receives the encoded data slices 376 andstores the encoded data slices 376 in one or more memory devices 374 ofthe plurality of memory devices 374. The DS processing module 372identifies encoded data slices of the stored encoded data slices 376 totransfer to produce identified encoded data slices 378. The DSprocessing module 372 identifies encoded data slices to transfer byidentifying a memory device 374 of the plurality of memory devices 374based on a memory device maintenance protocol (e.g., replace due to age,performance degradation, etc.) and identifying encoded data slicesstored by the memory device 374 as the identified encoded data slices378. Alternatively, the DS processing module 372 identifies theidentified encoded data slices 378 by determining that the identifiedencoded data slices 378 have a security level (e.g., a higher thanaverage security level requirement) that indicates a requirement fordata transfer.

The DS processing module 372 determines whether the other DS unit 36 hasanother local DSN address range in the address sector based on theglobal DSN address space. The DS processing module 372 determineswhether the other DS unit 36 has the other local DSN address range by atleast one of a variety of approaches. A first approach includesaccessing a mapping of the global DSN address space. A second approachincludes transmitting a request to a DSN management unit. A thirdapproach includes transmitting a global request to DS units of the DSN.

When the other DS unit 36 has the other local DSN address range in theaddress sector, the DS processing module 372 determines whether totransfer at least some of the identified encoded data slices to theother DS unit 36 based on a DSN data protocol. The DSN data protocolincludes one of a variety of approaches. A first approach includesdetermining DSN operational factors (e.g., a performance level, asecurity requirement, network traffic, availability of the other DSunit, capacity of the other DS unit, etc.) and, when the DSN operationalfactors are favorable, indicating that the identified encoded dataslices 378 are to be transferred. A second approach includes accessing apredetermined data transfer plan. A third approach includes querying DSunits of the DSN to identify the other DS unit 378. Alternatively, theDS processing 372 determines whether to transfer at least some of theidentified encoded data slices to the other DS unit 36 by determiningwhether the other DS unit 36 has a desired security level when the DSprocessing module 372 identifies the identified encoded data slices 378by determining that the identified encoded data slices 378 have thesecurity level that indicates the requirement for data transfer.

When the at least some of the identified encoded data slices are to betransferred to the other DS unit, the DS processing module 372 initiatesa data transfer protocol with the other DS unit. The DS processingmodule 372 initiates the data transfer protocol by a series of steps. Afirst step includes communicating with the other DS unit 36 to confirmtransferring of the at least some of the identified encoded data slices.When the transferring of the at least some of the identified encodeddata slices is confirmed, a second step includes transmitting a set ofwrite messages regarding the at least some of the identified encodeddata slices to other DS unit. The set of write messages is in accordancewith a DSN write protocol (e.g., a write slice request).

FIG. 17E is a flowchart illustrating another example of migratingslices. The method begins at step 380 or a processing module (e.g., of adispersed storage (DS) unit) identifies encoded data slices of storedencoded data slices to transfer to produce identified encoded dataslices. Data segments of data are encoded using a dispersed storageerror coding function to produce a plurality of sets of encoded dataslices. The encoded data slices includes one or more data slices of atleast some sets of the plurality of sets of encoded data slices. Thestored encoded data slices are assigned addresses within a localdistributed storage network (DSN) address range. A global DSN addressspace is divided into a plurality of address sectors. The local DSNaddress range is a portion of an address sector of the plurality ofaddress sectors. The plurality of sets of encoded data slices have aplurality of sets of slice names. A slice name of the plurality of setsof slices names includes an address sector field that identifies one ofthe plurality of address sectors, which corresponds to an ordering ofthe dispersed storage error coding function. The local DSN address rangeand another local DSN address range of another DS unit are assigned tothe address sector by a system management function.

The processing module identifies the encoded data slices to transfer bya series of steps. A first step includes identifying a memory device ofa plurality of memory devices based on a memory device maintenanceprotocol. A second step includes identifying encoded data slices storedby the memory device as the identified encoded data slices.Alternatively, the processing module identifies the identified encodeddata slices by determining that the identified encoded data slices havea security level that indicates a requirement for data transfer.

The method continues at step 382 where the processing module determineswhether the other local DSN address range in the address sector existsbased on the global DSN address space. The processing module determineswhether the other local DSN address range in the address sector existsby at least one of a variety of approaches. A first approach includesaccessing a mapping of the global DSN address space. A second approachincludes transmitting a request to a DSN management unit. A thirdapproach includes transmitting a global request to DS units of the DSN.

When the other local DSN address range in the address sector exists, themethod continues at step 384 where the processing module determineswhether to transfer at least some of the identified encoded data slicesinto the other local DSN address range based on a DSN data protocol. TheDSN data protocol includes one of a variety of approaches. A firstapproach includes determining DSN operational factors and, when the DSNoperational factors are favorable, indicating that the identifiedencoded data slices are to be transferred. A second approach includesaccessing a predetermined data transfer plan. A third approach includesquerying DS units of the DSN to identify the other local DSN addressrange. Alternatively, the processing module determines whether totransfer at least some of the identified encoded data slices to theother local DSN address range by determining whether the other local DSNaddress range has a desired security level when the processing moduleidentifies the identified encoded data slices by determining that theidentified encoded data slices have the security level that indicatesthe requirement for data transfer.

When the at least some of the identified encoded data slices are to betransferred, the method continues at step 386 where the processingmodule initiates a data transfer protocol to transfer the identifiedencoded data slices. The processing module initiates the data transferprotocol by a series of steps. A first step includes communicatingmessages to confirm transferring of the at least some of the identifiedencoded data slices. When the transferring of the at least some of theidentified encoded data slices is confirmed, a second step includestransmitting a set of write messages regarding the at least some of theidentified encoded data slices, wherein the set of write messages is inaccordance with a DSN write protocol.

FIG. 18A is a flowchart illustrating an example of sending a securemessage. The method begins at step 388 where a processing module (e.g.,of a dispersed storage (DS) module) encodes secret data using adispersed storage error coding function to produce a plurality of setsof encoded data slices. The secret data include sensitive data of anature to be shared on a need to know or permissions basis. For example,the secret data may include one or more of a confidential document, adigital certificate, an encryption key, network access information, apassword, personal information, etc.

The method continues at step 390 where the processing module generatesrecipient access information. The recipient access information includesone or more of a recipient identifier (ID), a public key associated withthe recipient, a certificate associated with the recipient, recipientvalidation information (e.g., a fingerprint based key), a message ID, amessage deletion policy, and a message access permissions. Thegeneration may be based on one or more of a requesting entity ID, alookup, receiving the access information, and a security levelrequirement.

The method continues at step 392 where the processing module appends therecipient access information to one or more encoded data slices of theplurality of sets of encoded data slices to produce a secure message.The method continues at step 394 where the processing module sends thesecure message to one or more intermediaries. The intermediary includesone or more of the user device, a DS processing unit, a DS unit, anintermediary server, and a DS managing unit. The sending includes one ormore of identifying the one or more intermediaries, selecting slices fortransmission to the one or more intermediaries (e.g., a portion of thesecure message), and outputting the slices for transmission to the oneor more intermediaries. The identifying the one or more intermediariesmay be based on one or more of the requesting entity ID, a recipient ID,a recipient ID to intermediary ID table lookup, and receiving one ormore identities corresponding to the one or more intermediaries.

FIG. 18B is a flowchart illustrating an example of processing a securemessage. The method begins at step 396 where a processing module (e.g.,of a dispersed storage (DS) module, of an intermediary) receives asecure message. The method continues at step 398 where the processingmodule receives a secure data request from a requesting entity (e.g.,from a user device). The request may include one or more of a requestingentity identifier (ID), a certificate, a public key, a secure messageID, and a secure message source ID.

The method continues at step 400 where the processing module verifiesthat the requesting entity is authorized to access the secure data. Theverifying includes one or more of indicating that the requesting entityis authorized to access the secure data when a recipient ID of thesecure data is substantially the same as the requesting entity ID andindicating that the requesting entity is authorized to access the securedata when an access control list lookup indicates that the requestingentity ID is authorized to access the secret data.

The method continues at step 402 where the processing moduleauthenticates the requesting entity when the requesting entity isauthorized. The authentication includes at least one of sending asignature request to the requesting entity, receiving a favorablesignature response, sending an encryption request to the requestingentity, and receiving a favorable encryption response.

The method continues at step 404 where the processing module facilitatessending one or more encoded data slices of the secure message to therequesting entity when the requesting entity is authenticated. Thesending includes one or more of selecting a portion of a plurality ofsets of encoded data slices of the secure message as the one or moreencoded data slices based on a portion indicator of recipientinformation extracted from the secure message.

The method continues at step 406 where the processing module deletes thesecure message when a delete message condition exists. The deletingincludes one or more of determining whether the delete message conditionexists and deleting at least some encoded data slices of the pluralityof sets of encoded data slices of the secure message. The determiningwhether the delete message condition exists includes one or more ofextracting a secure message deletion policy from the recipient accessinformation and indicating that the delete message condition exists whena delete message trigger is detected of the secure message deletionpolicy. The delete message trigger includes one or more of securemessage delivered to recipient, a time period has expired since thesecure message was delivered to the recipient, a time period has expiredsince the secure message was received, all messages delivered, and thedelivery threshold number of messages have been delivered.

FIG. 19A is a schematic block diagram of another embodiment of acomputing system that includes a dispersed storage (DS) unit 410. The DSunit 410 includes a DS processing module 412, a plurality of memorydevices 414, and a local memory device 416. The DS unit 410 functionsinclude one or more of storing encoded data slices, retrieving theencoded data slices, and deleting the encoded data slices. The DS unit410 receives a request 420 which includes at least one of a request toread at least a portion of corresponding encoded data slices, a requestto write the at least a portion of the corresponding encoded dataslices, and a request to delete the at least a portion of thecorresponding encoded data slices. The DS unit 410 processes the request420 and may generate a response 422 with regards to the processing ofthe request 420. The response 422 may include the at least a portion ofthe corresponding encoded data slices when the request 420 includes therequest to read the at least a portion of the corresponding encoded dataslices.

The plurality of memory devices 414 stores a plurality of collections ofencrypted and encoded data slices 428. A collection of encrypted andencoded data slices 428 of the plurality of collections of encrypted andencoded data slices 428 includes a common data aspect (e.g., belong tosame data object, same data file, same folder, same vault, same user ID,etc.). Encrypted and encoded data slices 428 of the collection ofencrypted and encoded data slices 428 are produced by individuallyencrypting corresponding encoded data slices using a common encryptingcharacter string 424 and representations 426 of the correspondingencoded data slices. The corresponding encoded data slices are dispersedstorage error encoded portions of a plurality of data segments (e.g., ofsame or different data objects, of same or different files, etc.). Forexample, a plurality of data segments of data (e.g., one or more dataobjects, one or more files, etc.) are each encoded (e.g., by a DSprocessing unit) using a dispersed storage error coding function toproduce a plurality of sets of encoded data slices. One or more encodeddata slices of at least some sets of the plurality of sets of encodeddata slices are selected as the corresponding encoded data slices (e.g.,slices associated with a common pillar number as a representation) andsent to the DS unit 410 for storage therein.

The common encrypting character string 424 includes one or more of arandom number, a series of characters, a passcode, and a random stringof characters. Each representation 426 of the representations 426 of thecorresponding encoded data slices includes one or more of distributedstorage network (DSN) address, a slice name of the corresponding encodeddata slice, a hash of an encoded data slice, the encoded data slice, anda vault ID.

The local memory 416 stores the common encrypting character string 424regarding the collection of encrypted and encoded data slices 428 andstores the representations 426 of the corresponding encoded data slices.For example, the DS processing module 412 generates a table to include aplurality of entries. Each entry of the plurality of entries includes aslice name, a corresponding storage location within the plurality ofmemory devices 414, and at least a portion of a corresponding commonencrypting character string 424. Next, the DS processing module 412stores the table in the local memory device 416. Alternatively, or inaddition to, the DS processing module 412 stores the table in adispersed storage network (DSN) memory.

The DS processing module 412 receives a request 420 regarding at least aportion of the corresponding encoded data slices. The DS processingmodule 412 identifies the common encrypting character string 424 of thecorresponding encoded data slices. For example, the DS processing module412 utilizes a slice name associated with the request 420 to access thetable from the local memory device 416 to recover the common encryptingcharacter string 424 of the corresponding encoded data slices.

When the request 420 is to delete the corresponding encoded data slices,the DS processing module 412 obfuscates the common encrypting characterstring 424 in the local memory such that the collection of encrypted andencoded data slices 428 are effectively incomprehensible. The processingmodule obfuscates the common encrypting character string 424 by at leastone of a variety of approaches. A first approach includes deleting thecommon encrypting character string 424. For example, the DS processingmodule 412 deletes the common encrypting character string 424 from thelocal memory 416. A second approach includes altering the commonencrypting character string 424. For example, the DS processing module412 alters the common encrypting character string 424 to include arandom number from a random number generator. A third approach includesmasking the common encrypting character string 424. The masking includesa series of steps. A first step includes transforming the commonencrypting character string 424 using a deterministic function and asecret key to produce a masked key. The deterministic function includesat least one of an exclusive OR logical function, subtraction, addition,a cyclic redundancy check (CRC), a hashing function, a hash basedmessage authentication code (HMAC) function, and a mask generatingfunction (MGF). For example, the DS processing module 412 performs anexclusive OR function on the common encrypting character string 424 andthe secret key to produce the masked key. The second step includesoverwriting the common encrypting character string 424 with the maskedkey.

When the request 420 is to read the at least a portion of thecorresponding encoded data slices, the DS processing module 412retrieves the common encrypting character string 424 and therepresentations 426 of the corresponding encoded data slices from thelocal memory device 416. Next, the DS processing module 412 decryptseach of at least a portion of the collection of encrypted and encodeddata slices 428 using the common encrypting character string 424 and acorresponding one of the representations 426 of the correspondingencoded data slices to recapture the at least a portion of thecorresponding encoded data slices. The DS processing module 412 outputsthe recaptured at least a portion of the corresponding encoded dataslices. For example, the DS processing module 412 generates the response422 to include the recaptured at least a portion of the correspondingencoded data slices and outputs the response 422.

When the request 420 is to write the at least a portion of thecorresponding encoded data slices, the DS processing module 412determines whether the collection of encrypted and encoded data slices428 exists. The determining may be based on accessing the table from thelocal memory device 416 and identifying whether a common encryptingcharacter string 424 exists that corresponds to a slice name of therequest to write the at least a portion of the corresponding encodeddata slices. The DS processing module 412 determines that the collectionof encrypted in encoded data slices 428 exists when the commonencrypting character string 424 exists that corresponds to the slicename of the request 420.

When the collection of encrypted and encoded data slices 428 exists, theDS processing module 412 retrieves the common encrypting characterstring 424 (e.g., if not already) and encrypts the at least a portion ofthe corresponding encoded data slices using the common encryptingcharacter string 424 and the representations 426 of the at least aportion of the corresponding encoded data slices to produce at least aportion of the collection of encrypted and encoded data slices 428.Next, the DS processing module 412 writes the at least a portion of thecollection of encrypted and encoded data slices 428 to one or more ofthe plurality of memory devices 414. The DS processing module 412individually encrypts the corresponding encoded data slices by a seriesof steps. A first step includes transforming the common encryptingcharacter string 424 and a representation (e.g., a slice name) ofrepresentations 426 of the corresponding encoded data slices using adeterministic function (e.g., exclusive OR) to produce an individualencryption key. A second step includes encrypting an individual encodeddata slice of the corresponding encoded data slices using the individualencryption key to produce an individual encrypted and encoded data sliceof the collection of encrypted and encoded data slices 428.

When the collection of encrypted and encoded data slices 428 does notexist, the DS processing module 412 generates the common encryptingcharacter string 424. For example, the DS processing module 412generates a new random number as the common encrypting character string424 corresponding to a vault identifier associated with the at least aportion of the corresponding encoded data slices. Next, the DSprocessing module 412 stores the common encrypting character string 424and the local memory device 416. The DS processing module 412 encryptsthe at least a portion of the corresponding encoded data slices usingthe common encrypting character string 424 and the representations 426of the at least a portion of the corresponding encoded data slices toproduce at least a portion of the collection of encrypted and encodeddata slices 428. Next, the DS processing module 412 writes the at leasta portion of the collection of encrypted and encoded data slices 428 toone or more of the plurality of memory devices 414.

FIG. 19B is a flowchart illustrating an example of deleting slices. Themethod begins at step 430 where a processing module (e.g., of adispersed storage (DS) unit) receives a request regarding at least aportion of corresponding encoded data slices. The request may include atleast one of a delete request, a read request, and a write request. Acollection of encrypted and encoded data slices of a plurality ofcollections of encrypted and encoded data slices includes a common dataaspect. Encrypted and encoded data slices of the collection of encryptedand encoded data slices are produced by individually encryptingcorresponding encoded data slices using a common encrypting characterstring and representations of the corresponding encoded data slices. Thecorresponding encoded data slices are dispersed storage error encodedportions of a plurality of data segments. The common encryptingcharacter string includes one or more of a random number, a series ofcharacters, a passcode, and a random string of characters. Eachrepresentation of the representations of the corresponding encoded dataslices includes one or more of distributed storage network (DSN)address, a slice name of a corresponding encoded data slice, a hash ofan encoded data slice, the encoded data slice, and a vault ID.

The method continues at step 432 where the processing module identifiesthe common encrypting character string of the corresponding encoded dataslices. When the request is to delete the corresponding encoded dataslices, the method continues at step 434 where the processing moduleobfuscates the common encrypting character string in a local memory suchthat the collection of encrypted and encoded data slices are effectivelyincomprehensible. The processing module obfuscates the common encryptingcharacter string by at least one of a variety of approaches. A firstapproach includes deleting the common encrypting character string. Asecond approach includes altering the common encrypting characterstring. A third approach includes that masking the common encryptingcharacter string. The masking includes a series of steps. A first stepincludes transforming the common encrypting character string using adeterministic function and a secret key to produce a masked key. Asecond step includes that overwriting the common encrypting characterstring with the masked key.

When the request is to read the at least a portion of the correspondingencoded data slices, the method continues at step 436 where theprocessing module retrieves the common encrypting character string andthe representations of the corresponding encoded data slices from thelocal memory. The method continues at step 438 where the processingmodule decrypts each of at least a portion of the collection ofencrypted and encoded data slices using the common encrypting characterstring and a corresponding one of the representations of thecorresponding encoded data slices to recapture the at least a portion ofthe corresponding encoded data slices. The method continues at step 440where the processing module outputs the recaptured at least a portion ofthe corresponding encoded data slices.

When the request is to write the at least a portion of the correspondingencoded data slices, the method continues at step 442 where theprocessing module determines whether the collection of encrypted andencoded data slices exists. The method branches to step 446 when thecollection of encrypted and encoded data slices exists. The methodcontinues to step 444 when the collection of encrypted encoded dataslices does not exist. The method continues at step 444 where theprocessing module generates the common encrypting character string. Whenthe collection of encrypted and encoded data slices exists, the methodcontinues at step 446 where the processing module retrieves the commonencrypting character string.

The method continues at step 448 where the processing module encryptsthe at least a portion of the corresponding encoded data slices usingthe common encrypting character string and the representations of the atleast a portion of the corresponding encoded data slices to produce atleast a portion of the collection of encrypted and encoded data slices.The processing module individually encrypts the corresponding encodeddata slices by a series of steps. A first step includes transforming thecommon encrypting character string and a representation ofrepresentations of the corresponding encoded data slices using adeterministic function to produce an individual encryption key. A secondstep includes encrypting an individual encoded data slice of thecorresponding encoded data slices using the individual encryption key toproduce an individual encrypted and encoded data slice of the collectionof encrypted and encoded data slices. The method continues at step 450where the processing module writes the at least a portion of thecollection of encrypted and encoded data slices to one or more memorydevices of a plurality of memory devices.

FIG. 19C is a schematic block diagram of another embodiment of acomputing system that includes a dispersed storage (DS) unit 460. The DSunit 460 includes a DS processing module 462, a plurality of memorydevices 464, and a local memory device 416. The plurality of memorydevices 464 includes two or more memory devices 414. The DS unit 462functions include one or more of storing encoded data slices, retrievingthe encoded data slices, and deleting the encoded data slices. The DSunit 462 receives a request 474 which includes at least one of a requestto read at least a portion of corresponding encoded data slices, arequest to write the at least a portion of the corresponding encodeddata slices, and a request to delete the at least a portion of thecorresponding encoded data slices. The request to delete the at least aportion of the corresponding encoded data slices includes at least oneof a first tier delete request and a second tier delete request. The DSunit 460 processes the request 474 and may generate a response 476 withregards to the processing of the request 474. The response 476 mayinclude the at least a portion of the corresponding encoded data sliceswhen the request 474 includes the request to read the at least a portionof the corresponding encoded data slices.

The plurality of memory devices 464 stores a plurality of encoded dataslices as a plurality of encrypted and encoded data slices. A pluralityof data segments is encoded (e.g., by a DS processing unit) using adispersed storage error encoding function to produce a plurality of setsof encoded data slices. The plurality of encoded data slices is a subset(e.g., a common pillar, one or more pillars, one or more encoded blocks,etc.) of the plurality of sets of encoded data slices. Groups of theplurality of encoded data slices are arranged into first tiercollections (e.g., by file or data object) of slices 466 based on commonfirst tier data aspects (e.g., by file name). Groups of the first tiercollections are arranged into second tier collections (e.g., by folderscontaining a plurality of files) of slices 468 based on common secondtier data aspects (e.g., folder name). Encoded data slices of one of thesecond tier collections of slices are individually encrypted using firsttier common encrypting character strings 470 relating to the commonfirst tier data aspects of the group of first tier collections of slices466 in the second tier collection of slices 468, a second tier commonencrypting character string 472 relating to the common second tier dataaspect of the one of the second tier collections of slices 468, andrepresentations 426 of the encoded data slices (e.g., slice names).

The DS processing module 462 functions to individually encrypt anencoded data slice of the encoded data slices by a series of steps. Afirst step includes transforming the first tier common encryptingcharacter string 470, the second tier common encrypting character string472, and a representation 426 of the encoded data slice (e.g., a slicename) using a deterministic function to produce an individual encryptionkey. A second step includes encrypting the encoded data slice using theindividual encryption key to produce an individual encrypted and encodeddata slice of the one of the second tier collections of slices 468.

The local memory 416 stores the first tier common encrypting characterstrings 470 relating to the common first tier data aspects of the groupof first tier collections of slices 466 in the second tier collection ofslices 468, the second tier common encrypting character string 472relating to the common second tier data aspect of the one of the secondtier collections of slices 468, and the representations 426 of theencoded data slices. The DS processing module 462 receives the request474 regarding encoded data slices having a common data aspect. When therequest 474 is the second tier delete request, the DS processing module462 identifies the second tier common encrypting character string 472corresponding to a second tier collection of slices 468 having thecommon data aspect (e.g., a particular folder name). Next, the DSprocessing module 462 obfuscates the second tier common encryptingcharacter string 472 in the local memory 416 such that the second tiercollection of slices 468 is effectively incomprehensible. For example,the DS processing module 462 overwrites the second tier commonencrypting character string 472 with a random pattern.

When the request 474 is the first tier delete request, the DS processingmodule 462 identifies a first tier common encrypting character string470 corresponding to one of the first tier collections of slices 466 ofone of the second tier collections of slices 468 having the common dataaspect (e.g., a particular filename. Next, the DS processing module 462obfuscates the first tier common encrypting character string 470 in thelocal memory 416 such that the one of the first tier collections ofslices 466 is effectively incomprehensible while other first tiercollections of slices 466 (e.g., other files) of the one of the secondtier collections of slices 468 (e.g., within a common folder) aresubstantially unaffected. For example, the DS processing module 462overwrites the first tier common encrypting character string 470 with apredetermined pattern.

FIG. 19D is a flowchart illustrating another example of deleting slices.The method begins at step 480 where a processing module (e.g., of adispersed storage (DS) unit) receives a request regarding encoded dataslices having a common data aspect. Groups of a plurality of encodeddata slices are arranged into first tier collections of slices based oncommon first tier data aspects. The plurality of encoded data slices isa subset of a plurality of sets of encoded data slices. A plurality ofdata segments is encoded (e.g., by a DS processing unit) using adispersed storage error encoding function to produce the plurality ofsets of encoded data slices. Groups of the first tier collections arearranged into second tier collections of slices based on common secondtier data aspects.

Encoded data slices of one of the second tier collections of slices areindividually encrypted using first tier common encrypting characterstrings relating to the common first tier data aspects of the group offirst tier collections of slices in the second tier collection ofslices, a second tier common encrypting character string relating to thecommon second tier data aspect of the one of the second tier collectionsof slices, and representations of the encoded data slices. Theprocessing module individually encrypts an encoded data slice of theencoded data slices by a series of steps. A first step includestransforming the first tier common encrypting character string, thesecond tier common encrypting character string, and a representation ofthe encoded data slice using a deterministic function to produce anindividual encryption key. For example, a processing module performs ahashing function on each of the first tier common encrypting characterstring, the second tier common encrypting character string, and a slicename (e.g., representation) of the encoded data slice to produce a keydigest set and performs an exclusive or function on each key digest ofthe key digest that to produce the individual encryption key. A secondstep includes encrypting the encoded data slice using the individualencryption key to produce an individual encrypted and encoded data sliceof the one of the second tier collections of slices.

When the request is a first tier delete request, the method continues atstep 482 where the processing module identifies a first tier commonencrypting character string corresponding to one of the first tiercollections of slices of one of the second tier collections of sliceshaving the common data aspect. For example, the processing moduleretrieves the first tier common encrypting character string from a localmemory based on the common data aspect of the request. The methodcontinues at step 484 where the processing module obfuscates the firsttier common encrypting character string (e.g., in a local memory) suchthat the one of the first tier collections of slices is effectivelyincomprehensible while other first tier collections of slices of the oneof the second tier collections of slices are substantially unaffected.

When the request is a second tier delete request, the method continuesat step 486 where the processing module identifies the second tiercommon encrypting character string corresponding to a second tiercollection of slices having the common data aspect. The method continuesat step 488 where the processing module obfuscates the second tiercommon encrypting character string (e.g., in the local memory) such thatthe second tier collection of slices is effectively incomprehensible.

FIG. 20 is a flowchart illustrating an example of outputting data. Themethod begins at step 490 where a processing module (e.g. of a dispersedstorage (DS) module, of a DS processing unit) opens a file to be writtento a dispersed storage network (DSN) memory. The file may include one ormore encoded data slices produced from dispersed storage error encodingdata. The method continues at step 492 where the processing moduleallocates a local cache memory to the file. The allocating includesidentifying a size associated with the file and selecting the localcache memory to include a favorable amount of memory in comparison tothe size associated with the file. The method continues at step 494where the processing module establishes an open output stream to the DSNmemory. For example, the processing module establishes a remote channelvia a network between a process of a DS processing unit and a process ofa DS unit.

The method continues at step 496 where the processing module saves aportion of the data in the local cache memory. The saving includesselecting the portion of the data in accordance with a selectionapproach and storing the portion of the data in the local cache memory.The selection approach includes selecting the portion received within atime period, selecting a first predetermined section of the data, andselecting a section based on a network performance indicator.

The method continues at step 498 where the processing module determineswhether to send the portion of the data to the DSN memory. Thedetermining may be based on one or more of receiving a write positionindicator from the DSN memory, comparing the write position to anexpected write position (e.g., maintained locally in a DS processingunit), and determining to send the portion of the data to the DSN memorywhen the comparison is favorable (e.g., substantially the same). Themethod branches to step 504 when the processing module determines not tosend the portion of the data to the DSN memory. The method continues tostep 500 when the processing module determines to send the portion ofthe data to the DSN memory.

The method continues at step 500 where the processing module sends theportion of the data to the DSN memory when the processing moduledetermines to send the portion of the data to the DSN memory. Forexample, the processing module writes the data to the output stream. Themethod continues at step 502 where the processing module determineswhether all portions of the data have been sent to the DSN memory. Forexample, the processing module determines that all portions of the datahave been sent to the DSN memory when all portions have been streamed.The method branches to step 512 when the processing module determinesthat all portions of the data have been sent to the DSN memory. Themethod repeats back to step 496 where the processing module saves theportion of the data in the local cache memory to save a next portion ofthe data when the processing module determines that all portions of thedata have not been sent to the DSN memory.

The method continues at step 504 where the processing modulede-establishes the open output stream to the DSN memory when theprocessing module determines not to send the portion of the data to theDSN memory. Alternatively, or in addition to, the processing module maysend a cancellation request to the DSN memory (e.g., to avoid partiallywritten data from becoming visible). The method continues at step 506where the processing module saves subsequent portions of the data in thelocal cache memory (e.g., other portions available now, other portionsavailable sequentially in the future).

The method continues at step 508 where the processing modulere-establishes the open output stream to the DSN memory (e.g., aftersaving all portions of the data in the local cache memory). The methodcontinues at step 510 where the processing module retrieves the datafrom the cache memory and sends the data to the DSN memory (e.g., theprocessing module streams out the data). The method continues at step512 where the processing module de-establishes the open output stream tothe DSN memory (e.g., the remote channel is deactivated). The methodcontinues at step 514 where the processing module determines adisposition for the data in the local cache memory. The dispositionincludes one or more deleting the data now, deleting the data whenadditional memory resources are required, and deleting later. Thedetermining may be based on one or more of an expected retrievalfrequency of the data, a predetermination, a message, a query, and alookup.

FIG. 21 is a flowchart illustrating an example of inputting data inaccordance with the invention, which include similar steps to FIG. 20.The method begins at step 516 where a processing module (e.g. of adispersed storage (DS) module, of a DS processing unit) opens a file tobe read from a dispersed storage network (DSN) memory (e.g., initiates aprocess to retrieve and decode a plurality of sets of encoded dataslices to re-create the file). The method continues with step 492 ofFIG. 20 where the processing module allocates local cache memory to thefile. The method continues at step 520 where the processing moduleestablishes an open input stream from the DSN memory at a starting readposition. For example, the processing module establishes a remotechannel via a network between a process of a DS processing unit and aprocess of a DS unit, wherein the starting read position corresponds toa starting location of the file.

The method continues at step 522 where the processing module determineswhether to retrieve a portion of the data from the DSN memory. Thedetermining may be based on one or more of receiving a read positionindicator from the DSN memory, comparing the read position to anexpected read position (e.g., maintained locally in a DS processingunit), and determining to retrieve the portion of the data from the DSNmemory when the comparison is favorable (e.g., substantially the same).The method branches to step 524 when the processing module determines toretrieve the portion of the data from the DSN memory. The methodcontinues to step 528 when the processing module determines not toretrieve the portion of the data from the DSN memory. The methodcontinues at step 528 where processing module de-establishes the openinput stream from the DSN memory. The method continues at step 530 wherethe processing module re-establishes the open input stream from the DSNmemory at a current read position. The method continues to step 524.

The method continues at step 524 where the processing module retrievesthe portion of the data and stores in the local cache memory. Forexample, the processing module sends a read request through the openinput stream and receives a portion of the data via the input streamfrom the DSN memory. The method continues at step 526 where theprocessing module determines whether all portions of the data have beenretrieved from the DSN memory. For example, the processing moduledetermines that all portions of the data have been retrieved from theDSN memory when all portions have been received via the stream.

The method branches to step 532 when the processing module determinesthat all portions of the data have been retrieved from the DSN memory.The method repeats back to step 522 where the processing moduledetermines whether to retrieve a portion of the data from the DSN memoryfor a next portion of the data when the processing module determinesthat all portions of the data have not been retrieved from the DSNmemory. The method continues at step 532 where the processing modulede-establishes the open input stream from the DSN memory when theprocessing module determines that all portions of the data have beenretrieved from the DSN memory. The method continues with step 514 ofFIG. 20 where the processing module determines a disposition for thedata in the local cache memory.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “operably coupled to”, “coupled to”, and/or “coupling” includesdirect coupling between items and/or indirect coupling between items viaan intervening item (e.g., an item includes, but is not limited to, acomponent, an element, a circuit, and/or a module) where, for indirectcoupling, the intervening item does not modify the information of asignal but may adjust its current level, voltage level, and/or powerlevel. As may further be used herein, inferred coupling (i.e., where oneelement is coupled to another element by inference) includes direct andindirect coupling between two items in the same manner as “coupled to”.As may even further be used herein, the term “operable to” or “operablycoupled to” indicates that an item includes one or more of powerconnections, input(s), output(s), etc., to perform, when activated, oneor more its corresponding functions and may further include inferredcoupling to one or more other items. As may still further be usedherein, the term “associated with”, includes direct and/or indirectcoupling of separate items and/or one item being embedded within anotheritem. As may be used herein, the term “compares favorably”, indicatesthat a comparison between two or more items, signals, etc., provides adesired relationship. For example, when the desired relationship is thatsignal 1 has a greater magnitude than signal 2, a favorable comparisonmay be achieved when the magnitude of signal 1 is greater than that ofsignal 2 or when the magnitude of signal 2 is less than that of signal1.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “operably coupled to”, “coupled to”, and/or “coupling” includesdirect coupling between items and/or indirect coupling between items viaan intervening item (e.g., an item includes, but is not limited to, acomponent, an element, a circuit, and/or a module) where, for indirectcoupling, the intervening item does not modify the information of asignal but may adjust its current level, voltage level, and/or powerlevel. As may further be used herein, inferred coupling (i.e., where oneelement is coupled to another element by inference) includes direct andindirect coupling between two items in the same manner as “coupled to”.As may even further be used herein, the term “operable to” or “operablycoupled to” indicates that an item includes one or more of powerconnections, input(s), output(s), etc., to perform, when activated, oneor more its corresponding functions and may further include inferredcoupling to one or more other items. As may still further be usedherein, the term “associated with”, includes direct and/or indirectcoupling of separate items and/or one item being embedded within anotheritem. As may be used herein, the term “compares favorably”, indicatesthat a comparison between two or more items, signals, etc., provides adesired relationship. For example, when the desired relationship is thatsignal 1 has a greater magnitude than signal 2, a favorable comparisonmay be achieved when the magnitude of signal 1 is greater than that ofsignal 2 or when the magnitude of signal 2 is less than that of signal1.

As may also be used herein, the terms “processing module”, “processingcircuit”, and/or “processing unit” may be a single processing device ora plurality of processing devices. Such a processing device may be amicroprocessor, micro-controller, digital signal processor,microcomputer, central processing unit, field programmable gate array,programmable logic device, state machine, logic circuitry, analogcircuitry, digital circuitry, and/or any device that manipulates signals(analog and/or digital) based on hard coding of the circuitry and/oroperational instructions. The processing module, module, processingcircuit, and/or processing unit may be, or further include, memoryand/or an integrated memory element, which may be a single memorydevice, a plurality of memory devices, and/or embedded circuitry ofanother processing module, module, processing circuit, and/or processingunit. Such a memory device may be a read-only memory, random accessmemory, volatile memory, non-volatile memory, static memory, dynamicmemory, flash memory, cache memory, and/or any device that storesdigital information. Note that if the processing module, module,processing circuit, and/or processing unit includes more than oneprocessing device, the processing devices may be centrally located(e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that if the processing module, module, processing circuit,and/or processing unit implements one or more of its functions via astate machine, analog circuitry, digital circuitry, and/or logiccircuitry, the memory and/or memory element storing the correspondingoperational instructions may be embedded within, or external to, thecircuitry comprising the state machine, analog circuitry, digitalcircuitry, and/or logic circuitry. Still further note that, the memoryelement may store, and the processing module, module, processingcircuit, and/or processing unit executes, hard coded and/or operationalinstructions corresponding to at least some of the steps and/orfunctions illustrated in one or more of the Figures. Such a memorydevice or memory element can be included in an article of manufacture.

The present invention has been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claimed invention. Further, theboundaries of these functional building blocks have been arbitrarilydefined for convenience of description. Alternate boundaries could bedefined as long as the certain significant functions are appropriatelyperformed. Similarly, flow diagram blocks may also have been arbitrarilydefined herein to illustrate certain significant functionality. To theextent used, the flow diagram block boundaries and sequence could havebeen defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claimed invention. One of average skill in the artwill also recognize that the functional building blocks, and otherillustrative blocks, modules and components herein, can be implementedas illustrated or by discrete components, application specificintegrated circuits, processors executing appropriate software and thelike or any combination thereof.

The present invention may have also been described, at least in part, interms of one or more embodiments. An embodiment of the present inventionis used herein to illustrate the present invention, an aspect thereof, afeature thereof, a concept thereof, and/or an example thereof. Aphysical embodiment of an apparatus, an article of manufacture, amachine, and/or of a process that embodies the present invention mayinclude one or more of the aspects, features, concepts, examples, etc.described with reference to one or more of the embodiments discussedherein. Further, from figure to figure, the embodiments may incorporatethe same or similarly named functions, steps, modules, etc. that may usethe same or different reference numbers and, as such, the functions,steps, modules, etc. may be the same or similar functions, steps,modules, etc. or different ones.

While the transistors in the above described figure(s) is/are shown asfield effect transistors (FETs), as one of ordinary skill in the artwill appreciate, the transistors may be implemented using any type oftransistor structure including, but not limited to, bipolar, metal oxidesemiconductor field effect transistors (MOSFET), N-well transistors,P-well transistors, enhancement mode, depletion mode, and zero voltagethreshold (VT) transistors.

Unless specifically stated to the contra, signals to, from, and/orbetween elements in a figure of any of the figures presented herein maybe analog or digital, continuous time or discrete time, and single-endedor differential. For instance, if a signal path is shown as asingle-ended path, it also represents a differential signal path.Similarly, if a signal path is shown as a differential path, it alsorepresents a single-ended signal path. While one or more particulararchitectures are described herein, other architectures can likewise beimplemented that use one or more data buses not expressly shown, directconnectivity between elements, and/or indirect coupling between otherelements as recognized by one of average skill in the art.

The term “module” is used in the description of the various embodimentsof the present invention. A module includes a processing module, afunctional block, hardware, and/or software stored on memory forperforming one or more functions as may be described herein. Note that,if the module is implemented via hardware, the hardware may operateindependently and/or in conjunction software and/or firmware. As usedherein, a module may contain one or more sub-modules, each of which maybe one or more modules.

While particular combinations of various functions and features of thepresent invention have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent invention is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A method comprises: receiving a file creationrequest to create a new file from portions of a set of existing files,wherein an existing file of the set of existing files includes aplurality of data segments that is stored as a plurality of sets ofencoded data slices in a dispersed storage network (DSN) memory;identifying data segments from the plurality of data segments of the setof existing files corresponding to the portions of the set of existingfiles to produce identified data segments; identifying sets of encodeddata slices of the plurality of sets of encoded data slices of theidentified data segments to produce identified sets of encoded dataslices; determining addressing information for the identified sets ofencoded data slices from existing addressing information of the set ofexisting files; and generating a segment allocation table for the newfile based on the addressing information of the identified sets ofencoded data slices such that the new file is created withoutduplication of the portions of the existing files.
 2. The method ofclaim 1 further comprises: the existing file of the set of existingfiles further includes a set of data regions, wherein a data region ofthe set of data regions includes a consecutive subset of the pluralityof data segments; identifying data regions from the set of data regionsof the set of existing files corresponding to the portions of the set ofexisting files to produce identified data regions; identifying the setsof encoded data slices of the plurality of sets of encoded data slicesof the identified data regions to produce the identified sets of encodeddata slices; and determining the addressing information for theidentified sets of encoded data slices from a set of segment allocationtables associated with the set of existing files, wherein the set ofsegment allocation tables includes the existing addressing informationof the set of existing files at data region level.
 3. The method ofclaim 2, wherein the identifying the sets of encoded data slicescomprises: retrieving a set of segment allocation tables for the set ofexisting files based on the file creation request, wherein a segmentallocation table of the set of segment allocation tables includes aplurality of entries, and wherein an entry of the plurality of entriesincludes a DSN address regarding storage of a data region of the set ofdata segments of the set of existing files in the DSN memory andsegmentation information regarding the dividing of the data region intothe consecutive subset of the plurality of data segments.
 4. The methodof claim 2, wherein addressing information for an identified set ofencoded data slices of the identified sets of encoded data slicescomprises: a start segment vault source name corresponding to a dataregion associated with the identified set of encoded data slices; andsegmentation information including one or more of: a segment size ofdata segments of the data region; a total length of the data region; anda segmentation approach of the data region.
 5. The method of claim 1,wherein the generating the segment allocation table for the new filecomprises: identifying consecutive data segments of the identified datasegments; grouping the consecutive data segments into a region; andgenerating addressing information for the region, which is stored as anentry in the segment allocation table, wherein individual addressinginformation of the identified sets of encoded data slices isdeterminable from the addressing information of the region.
 6. Themethod of claim 1, wherein the identifying data segments comprises atleast one of: receiving, within the file creation request,identification of the identified data segments; and searching, based ona search criteria, the set of existing files, or addressing informationthereof, to identify the data segments.
 7. The method of claim 1,wherein the identifying the sets of encoded data slices comprises:retrieving a set of segment allocation tables for the set of existingfiles based on the file creation request, wherein a segment allocationtable of the set of segment allocation tables includes a plurality ofentries, and wherein an entry of the plurality of entries includes a DSNaddress regarding storage of a data segment of the plurality of datasegments in the DSN memory.
 8. The method of claim 1 further comprises:obtaining a segment allocation table vault source name for the segmentallocation table; encoding the segment allocation table using adispersed storage error coding function to produce encoded table slices;and outputting the encoded table slices to the DSN memory for storagetherein utilizing the segment allocation table vault source name.
 9. Adispersed storage (DS) module comprises: a first module, when operablewithin a computing device, causes the computing device to: receive afile creation request to create a new file from portions of a set ofexisting files, wherein an existing file of the set of existing filesincludes a plurality of data segments that is stored as a plurality ofsets of encoded data slices in a dispersed storage network (DSN) memory;a second module, when operable within the computing device, causes thecomputing device to: identify data segments from the plurality of datasegments of the set of existing files corresponding to the portions ofthe set of existing files to produce identified data segments; andidentify sets of encoded data slices of the plurality of sets of encodeddata slices of the identified data segments to produce identified setsof encoded data slices; a third module, when operable within thecomputing device, causes the computing device to: determine addressinginformation for the identified sets of encoded data slices from existingaddressing information of the set of existing files; and a fourthmodule, when operable within the computing device, causes the computingdevice to: generate a segment allocation table for the new file based onthe addressing information of the identified sets of encoded data slicessuch that the new file is created without duplication of the portions ofthe existing files.
 10. The DS module of claim 9 further comprises: theexisting file of the set of existing files further includes a set ofdata regions, wherein a data region of the set of data regions includesa consecutive subset of the plurality of data segments; the secondmodule further functions to: identify data regions from the set of dataregions of the set of existing files corresponding to the portions ofthe set of existing files to produce identified data regions; andidentify the sets of encoded data slices of the plurality of sets ofencoded data slices of the identified data regions to produce theidentified sets of encoded data slices; and the third module furtherfunctions to determine the addressing information for the identifiedsets of encoded data slices from a set of segment allocation tablesassociated with the set of existing files, wherein the set of segmentallocation tables includes the existing addressing information of theset of existing files at data region level.
 11. The DS module of claim10, wherein the second module functions to identify the sets of encodeddata slices by: retrieving a set of segment allocation tables for theset of existing files based on the file creation request, wherein asegment allocation table of the set of segment allocation tablesincludes a plurality of entries, and wherein an entry of the pluralityof entries includes a DSN address regarding storage of a data region ofthe set of data segments of the set of existing files in the DSN memoryand segmentation information regarding the dividing of the data regioninto the consecutive subset of the plurality of data segments.
 12. TheDS module of claim 10, wherein addressing information for an identifiedset of encoded data slices of the identified sets of encoded data slicescomprises: a start segment vault source name corresponding to a dataregion associated with the identified set of encoded data slices; andsegmentation information including one or more of: a segment size ofdata segments of the data region; a total length of the data region; anda segmentation approach of the data region.
 13. The DS module of claim9, wherein the fourth module functions to generate the segmentallocation table for the new file by: identifying consecutive datasegments of the identified data segments; grouping the consecutive datasegments into a region; and generating addressing information for theregion, which is stored as an entry in the segment allocation table,wherein individual addressing information of the identified sets ofencoded data slices is determinable from the addressing information ofthe region.
 14. The DS module of claim 9, wherein the second modulefunctions to identify data segments by at least one of: receiving,within the file creation request, identification of the identified datasegments; and searching, based on a search criteria, the set of existingfiles, or addressing information thereof, to identify the data segments.15. The DS module of claim 9, wherein the second module functions toidentify the sets of encoded data slices by: retrieving a set of segmentallocation tables for the set of existing files based on the filecreation request, wherein a segment allocation table of the set ofsegment allocation tables includes a plurality of entries, and whereinan entry of the plurality of entries includes a DSN address regardingstorage of a data segment of the plurality of data segments in the DSNmemory.
 16. The DS module of claim 9 further comprises: the fourthmodule further functions to: obtain a segment allocation table vaultsource name for the segment allocation table; encode the segmentallocation table using a dispersed storage error coding function toproduce encoded table slices; and output the encoded table slices to theDSN memory for storage therein utilizing the segment allocation tablevault source name.